* [RFC PATCH 0/9] Ksplice: Rebootless kernel updates @ 2008-09-13 4:49 Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 1/9] Ksplice: Make find_symbol return a struct kernel_symbol Jeffrey Brian Arnold 2008-09-13 5:12 ` [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Alexey Dobriyan 0 siblings, 2 replies; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko These RFC patches add support for Ksplice [1], a rebootless update system, to the kernel. I previously mentioned Ksplice to the LKML a few months ago [2]. The basic idea is that Ksplice can apply an update to a running Linux kernel based only on a source code patch and the original kernel source. The Ksplice technical overview document [3] describes the design of Ksplice and one of the ways that Ksplice has been evaluated. [1] http://web.mit.edu/ksplice [2] http://lkml.org/lkml/2008/4/23/330 [3] http://web.mit.edu/ksplice/doc/ksplice.pdf The interface between the Ksplice kernel component and the userspace utilities is documented in patch 6/9 of this series. The current implementation of the userspace utilities is available from the Ksplice website and the Ksplice git repository [4]. The Ksplice userspace utilities transparently support both "integrated" operation (updating a kernel patched with this patch series) and "standalone" operation (updating a completely unmodified kernel). [4] http://web.mit.edu/ksplice/ksplice.git This series of patches depends upon the -ffunction-sections and -fdata-sections patches proposed by Denys Vlasenko (for reasons unrelated to Ksplice) a few weeks ago [5]. Ksplice's 'run-pre matching' process is much simpler if the original kernel was compiled with -ffunction-sections and -fdata-sections, and so we require these compiler options for this patch series. We think that requiring these options in order to build a kernel with integrated Ksplice support is reasonable, but we welcome feedback. [5] http://lkml.org/lkml/2008/8/24/159 Denys Vlasenko has already submitted patches to the LKML for allowing all architectures to compile with -ffunction-sections and -fdata-sections [5]; he has said that he will later submit patches making it possible to run a kernel with these options [6]. In the meantime, we have a patch that creates a CONFIG_FUNCTION_DATA_SECTIONS option for x86 and ARM [7], but we have not polished this patch since Denys Vlasenko is already preparing a similar change. [6] http://lkml.org/lkml/2008/8/26/288 [7] http://web.mit.edu/ksplice/tmp/function-data-sections.patch We're open to suggestions and are available to make whatever changes people would like to see. (Also, I'll be at the Plumbers Conference next week and would be happy to chat with anyone about Ksplice in person). Jeff Arnold jbarnold@mit.edu -- Documentation/ksplice.txt | 247 +++++ MAINTAINERS | 10 + arch/Kconfig | 14 + arch/arm/Kconfig | 1 + arch/arm/kernel/ksplice-arch.c | 79 ++ arch/x86/Kconfig | 1 + arch/x86/kernel/ksplice-arch.c | 91 ++ arch/x86/mm/init_32.c | 7 +- arch/x86/mm/init_64.c | 5 +- include/linux/kallsyms.h | 13 + include/linux/ksplice.h | 202 ++++ include/linux/module.h | 27 + kernel/Makefile | 2 + kernel/kallsyms.c | 19 + kernel/ksplice.c | 1997 ++++++++++++++++++++++++++++++++++++++++ kernel/module.c | 144 ++- kernel/sched.c | 1 + 17 files changed, 2810 insertions(+), 50 deletions(-) ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 1/9] Ksplice: Make find_symbol return a struct kernel_symbol 2008-09-13 4:49 [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 2/9] Ksplice: Add module_data_address (the analogue of module_text_address) Jeffrey Brian Arnold 2008-09-13 5:12 ` [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Alexey Dobriyan 1 sibling, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko From: Tim Abbott <tabbott@mit.edu> Ksplice needs access to the kernel_symbol structure in order to support modifications to the exported symbol table. Signed-off-by: Tim Abbott <tabbott@mit.edu> --- kernel/module.c | 80 ++++++++++++++++++++++++++---------------------------- 1 files changed, 39 insertions(+), 41 deletions(-) diff --git a/kernel/module.c b/kernel/module.c index 6cedfc7..8954556 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -254,7 +254,7 @@ struct find_symbol_arg { /* Output */ struct module *owner; const unsigned long *crc; - unsigned long value; + const struct kernel_symbol *sym; }; static bool find_symbol_in_section(const struct symsearch *syms, @@ -295,17 +295,17 @@ static bool find_symbol_in_section(const struct symsearch *syms, fsa->owner = owner; fsa->crc = symversion(syms->crcs, symnum); - fsa->value = syms->start[symnum].value; + fsa->sym = &syms->start[symnum]; return true; } -/* Find a symbol, return value, (optional) crc and (optional) module - * which owns it */ -static unsigned long find_symbol(const char *name, - struct module **owner, - const unsigned long **crc, - bool gplok, - bool warn) +/* Find a symbol and return it, along with, (optional) crc and + * (optional) module which owns it */ +static const struct kernel_symbol *find_symbol(const char *name, + struct module **owner, + const unsigned long **crc, + bool gplok, + bool warn) { struct find_symbol_arg fsa; @@ -318,11 +318,11 @@ static unsigned long find_symbol(const char *name, *owner = fsa.owner; if (crc) *crc = fsa.crc; - return fsa.value; + return fsa.sym; } DEBUGP("Failed to find symbol %s\n", name); - return -ENOENT; + return NULL; } /* Search for module by name: must hold module_mutex. */ @@ -819,7 +819,7 @@ void __symbol_put(const char *symbol) struct module *owner; preempt_disable(); - if (IS_ERR_VALUE(find_symbol(symbol, &owner, NULL, true, false))) + if (find_symbol(symbol, &owner, NULL, true, false) == NULL) BUG(); module_put(owner); preempt_enable(); @@ -982,7 +982,7 @@ static inline int check_modstruct_version(Elf_Shdr *sechdrs, { const unsigned long *crc; - if (IS_ERR_VALUE(find_symbol("struct_module", NULL, &crc, true, false))) + if (find_symbol("struct_module", NULL, &crc, true, false) == NULL) BUG(); return check_version(sechdrs, versindex, "struct_module", mod, crc); } @@ -1023,25 +1023,23 @@ static inline int same_magic(const char *amagic, const char *bmagic, /* Resolve a symbol for this module. I.e. if we find one, record usage. Must be holding module_mutex. */ -static unsigned long resolve_symbol(Elf_Shdr *sechdrs, - unsigned int versindex, - const char *name, - struct module *mod) +static const struct kernel_symbol *resolve_symbol(Elf_Shdr *sechdrs, + unsigned int versindex, + const char *name, + struct module *mod) { struct module *owner; - unsigned long ret; + const struct kernel_symbol *sym; const unsigned long *crc; - ret = find_symbol(name, &owner, &crc, + sym = find_symbol(name, &owner, &crc, !(mod->taints & TAINT_PROPRIETARY_MODULE), true); - if (!IS_ERR_VALUE(ret)) { - /* use_module can fail due to OOM, - or module initialization or unloading */ - if (!check_version(sechdrs, versindex, name, mod, crc) || - !use_module(mod, owner)) - ret = -EINVAL; - } - return ret; + /* use_module can fail due to OOM, + or module initialization or unloading */ + if (!check_version(sechdrs, versindex, name, mod, crc) || + !use_module(mod, owner)) + sym = NULL; + return sym; } /* @@ -1445,17 +1443,15 @@ static void free_module(struct module *mod) void *__symbol_get(const char *symbol) { struct module *owner; - unsigned long value; + const struct kernel_symbol *sym; preempt_disable(); - value = find_symbol(symbol, &owner, NULL, true, true); - if (IS_ERR_VALUE(value)) - value = 0; - else if (strong_try_module_get(owner)) - value = 0; + sym = find_symbol(symbol, &owner, NULL, true, true); + if (sym != NULL && strong_try_module_get(owner)) + sym = NULL; preempt_enable(); - return (void *)value; + return sym == NULL ? (void *)0 : (void *)sym->value; } EXPORT_SYMBOL_GPL(__symbol_get); @@ -1483,8 +1479,8 @@ static int verify_export_symbols(struct module *mod) for (i = 0; i < ARRAY_SIZE(arr); i++) { for (s = arr[i].sym; s < arr[i].sym + arr[i].num; s++) { - if (!IS_ERR_VALUE(find_symbol(s->name, &owner, - NULL, true, false))) { + if (find_symbol(s->name, &owner, NULL, true, false) + != NULL) { printk(KERN_ERR "%s: exports duplicate symbol %s" " (owned by %s)\n", @@ -1508,6 +1504,7 @@ static int simplify_symbols(Elf_Shdr *sechdrs, unsigned long secbase; unsigned int i, n = sechdrs[symindex].sh_size / sizeof(Elf_Sym); int ret = 0; + const struct kernel_symbol *ksym; for (i = 1; i < n; i++) { switch (sym[i].st_shndx) { @@ -1527,13 +1524,14 @@ static int simplify_symbols(Elf_Shdr *sechdrs, break; case SHN_UNDEF: - sym[i].st_value - = resolve_symbol(sechdrs, versindex, - strtab + sym[i].st_name, mod); - + ksym = resolve_symbol(sechdrs, versindex, + strtab + sym[i].st_name, mod); /* Ok if resolved. */ - if (!IS_ERR_VALUE(sym[i].st_value)) + if (ksym != NULL) { + sym[i].st_value = ksym->value; break; + } + /* Ok if weak. */ if (ELF_ST_BIND(sym[i].st_info) == STB_WEAK) break; -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 2/9] Ksplice: Add module_data_address (the analogue of module_text_address) 2008-09-13 4:49 ` [RFC PATCH 1/9] Ksplice: Make find_symbol return a struct kernel_symbol Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 3/9] Ksplice: Add functions for walking kallsyms symbols Jeffrey Brian Arnold 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko From: Tim Abbott <tabbott@mit.edu> Ksplice's run-pre matching process needs to be able to determine the module that contains a particular data address. Signed-off-by: Tim Abbott <tabbott@mit.edu> --- include/linux/module.h | 2 ++ kernel/module.c | 28 ++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 0 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index 68e0955..b7cdfb4 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -362,6 +362,8 @@ static inline int module_is_live(struct module *mod) /* Is this address in a module? (second is with no locks, for oops) */ struct module *module_text_address(unsigned long addr); struct module *__module_text_address(unsigned long addr); +struct module *module_data_address(unsigned long addr); +struct module *__module_data_address(unsigned long addr); int is_module_address(unsigned long addr); /* Returns 0 and fills in value, defined and namebuf, or -ERANGE if diff --git a/kernel/module.c b/kernel/module.c index 8954556..dda2f80 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -2682,6 +2682,34 @@ struct module *module_text_address(unsigned long addr) return mod; } +struct module *__module_data_address(unsigned long addr) +{ + struct module *mod; + + if (addr < module_addr_min || addr > module_addr_max) + return NULL; + + list_for_each_entry(mod, &modules, list) { + if (within(addr, mod->module_core + mod->core_text_size, + mod->core_size - mod->core_text_size) || + within(addr, mod->module_init + mod->init_text_size, + mod->init_size - mod->init_text_size)) + return mod; + } + return NULL; +} + +struct module *module_data_address(unsigned long addr) +{ + struct module *mod; + + preempt_disable(); + mod = __module_data_address(addr); + preempt_enable(); + + return mod; +} + /* Don't grab lock, we're oopsing. */ void print_modules(void) { -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 3/9] Ksplice: Add functions for walking kallsyms symbols 2008-09-13 4:49 ` [RFC PATCH 2/9] Ksplice: Add module_data_address (the analogue of module_text_address) Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 4/9] Ksplice: Make needed symbol and module functions global Jeffrey Brian Arnold 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko From: Anders Kaseorg <andersk@mit.edu> kallsyms_lookup_name only returns the first match that it finds. Ksplice needs information about all symbols with a given name in order to correctly resolve local symbols. kallsyms_on_each_symbol provides a generic mechanism for iterating over the kallsyms table. Signed-off-by: Anders Kaseorg <andersk@mit.edu> --- include/linux/kallsyms.h | 13 +++++++++++++ include/linux/module.h | 12 ++++++++++++ kernel/kallsyms.c | 18 ++++++++++++++++++ kernel/module.c | 19 +++++++++++++++++++ 4 files changed, 62 insertions(+), 0 deletions(-) diff --git a/include/linux/kallsyms.h b/include/linux/kallsyms.h index b961448..b8ee4d2 100644 --- a/include/linux/kallsyms.h +++ b/include/linux/kallsyms.h @@ -17,6 +17,11 @@ /* Lookup the address for a symbol. Returns 0 if not found. */ unsigned long kallsyms_lookup_name(const char *name); +/* Call a function on each kallsyms symbol in the core kernel */ +int kallsyms_on_each_symbol(int (*fn)(void *, const char *, struct module *, + unsigned long), + void *data); + extern int kallsyms_lookup_size_offset(unsigned long addr, unsigned long *symbolsize, unsigned long *offset); @@ -43,6 +48,14 @@ static inline unsigned long kallsyms_lookup_name(const char *name) return 0; } +static inline int kallsyms_on_each_symbol(int (*fn)(void *, const char *, + struct module *, + unsigned long), + void *data) +{ + return 0; +} + static inline int kallsyms_lookup_size_offset(unsigned long addr, unsigned long *symbolsize, unsigned long *offset) diff --git a/include/linux/module.h b/include/linux/module.h index b7cdfb4..cd982cd 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -374,6 +374,10 @@ int module_get_kallsym(unsigned int symnum, unsigned long *value, char *type, /* Look for this name: can be of form module:name. */ unsigned long module_kallsyms_lookup_name(const char *name); +int module_kallsyms_on_each_symbol(int (*fn)(void *, const char *, + struct module *, unsigned long), + void *data); + extern void __module_put_and_exit(struct module *mod, long code) __attribute__((noreturn)); #define module_put_and_exit(code) __module_put_and_exit(THIS_MODULE, code); @@ -539,6 +543,14 @@ static inline unsigned long module_kallsyms_lookup_name(const char *name) return 0; } +static inline int module_kallsyms_on_each_symbol(int (*fn)(void *, const char *, + struct module *, + unsigned long), + void *data) +{ + return 0; +} + static inline int register_module_notifier(struct notifier_block * nb) { /* no events will happen anyway, so this can always succeed */ diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 38fc10a..47d3e94 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -161,6 +161,24 @@ unsigned long kallsyms_lookup_name(const char *name) return module_kallsyms_lookup_name(name); } +int kallsyms_on_each_symbol(int (*fn)(void *, const char *, struct module *, + unsigned long), + void *data) +{ + char namebuf[KSYM_NAME_LEN]; + unsigned long i; + unsigned int off; + int ret; + + for (i = 0, off = 0; i < kallsyms_num_syms; i++) { + off = kallsyms_expand_symbol(off, namebuf); + ret = fn(data, namebuf, NULL, kallsyms_addresses[i]); + if (ret != 0) + return ret; + } + return module_kallsyms_on_each_symbol(fn, data); +} + static unsigned long get_symbol_pos(unsigned long addr, unsigned long *symbolsize, unsigned long *offset) diff --git a/kernel/module.c b/kernel/module.c index dda2f80..343c8f8 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -2523,6 +2523,25 @@ unsigned long module_kallsyms_lookup_name(const char *name) preempt_enable(); return ret; } + +int module_kallsyms_on_each_symbol(int (*fn)(void *, const char *, + struct module *, unsigned long), + void *data) +{ + struct module *mod; + unsigned int i; + int ret; + + list_for_each_entry(mod, &modules, list) { + for (i = 0; i < mod->num_symtab; i++) { + ret = fn(data, mod->strtab + mod->symtab[i].st_name, + mod, mod->symtab[i].st_value); + if (ret != 0) + return ret; + } + } + return 0; +} #endif /* CONFIG_KALLSYMS */ /* Called by the /proc file system to return a list of modules. */ -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 4/9] Ksplice: Make needed symbol and module functions global 2008-09-13 4:49 ` [RFC PATCH 3/9] Ksplice: Add functions for walking kallsyms symbols Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 5/9] Ksplice: Architecture-independent core Jeffrey Brian Arnold 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko From: Tim Abbott <tabbott@mit.edu> Make module_mutex, find_module(), find_symbol(), and use_module() global. Ksplice needs access to these functions from module.c in order to resolve symbols and implement dependency handling. Signed-off-by: Tim Abbott <tabbott@mit.edu> --- include/linux/module.h | 13 +++++++++++++ kernel/module.c | 18 +++++++++--------- 2 files changed, 22 insertions(+), 9 deletions(-) diff --git a/include/linux/module.h b/include/linux/module.h index cd982cd..6ae0fc9 100644 --- a/include/linux/module.h +++ b/include/linux/module.h @@ -351,6 +351,8 @@ struct module #define MODULE_ARCH_INIT {} #endif +extern struct mutex module_mutex; + /* FIXME: It'd be nice to isolate modules during init, too, so they aren't used before they (may) fail. But presently too much code (IDE & SCSI) require entry into the module during init.*/ @@ -366,6 +368,16 @@ struct module *module_data_address(unsigned long addr); struct module *__module_data_address(unsigned long addr); int is_module_address(unsigned long addr); +/* Search for module by name: must hold module_mutex. */ +struct module *find_module(const char *name); + +/* Search for an exported symbol by name. */ +const struct kernel_symbol *find_symbol(const char *name, + struct module **owner, + const unsigned long **crc, + bool gplok, + bool warn); + /* Returns 0 and fills in value, defined and namebuf, or -ERANGE if symnum out of range. */ int module_get_kallsym(unsigned int symnum, unsigned long *value, char *type, @@ -431,6 +443,7 @@ static inline void __module_get(struct module *module) #define symbol_put_addr(p) do { } while(0) #endif /* CONFIG_MODULE_UNLOAD */ +int use_module(struct module *a, struct module *b); /* This is a #define so the string doesn't get put in every .o file */ #define module_name(mod) \ diff --git a/kernel/module.c b/kernel/module.c index 343c8f8..2acb646 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -62,7 +62,7 @@ /* List of modules, protected by module_mutex or preempt_disable * (add/delete uses stop_machine). */ -static DEFINE_MUTEX(module_mutex); +DEFINE_MUTEX(module_mutex); static LIST_HEAD(modules); /* Waiting for a module to finish initializing? */ @@ -301,11 +301,11 @@ static bool find_symbol_in_section(const struct symsearch *syms, /* Find a symbol and return it, along with, (optional) crc and * (optional) module which owns it */ -static const struct kernel_symbol *find_symbol(const char *name, - struct module **owner, - const unsigned long **crc, - bool gplok, - bool warn) +const struct kernel_symbol *find_symbol(const char *name, + struct module **owner, + const unsigned long **crc, + bool gplok, + bool warn) { struct find_symbol_arg fsa; @@ -326,7 +326,7 @@ static const struct kernel_symbol *find_symbol(const char *name, } /* Search for module by name: must hold module_mutex. */ -static struct module *find_module(const char *name) +struct module *find_module(const char *name) { struct module *mod; @@ -579,7 +579,7 @@ static int already_uses(struct module *a, struct module *b) } /* Module a uses b */ -static int use_module(struct module *a, struct module *b) +int use_module(struct module *a, struct module *b) { struct module_use *use; int no_warn, err; @@ -874,7 +874,7 @@ static inline void module_unload_free(struct module *mod) { } -static inline int use_module(struct module *a, struct module *b) +int use_module(struct module *a, struct module *b) { return strong_try_module_get(b) == 0; } -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 5/9] Ksplice: Architecture-independent core 2008-09-13 4:49 ` [RFC PATCH 4/9] Ksplice: Make needed symbol and module functions global Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold [not found] ` <8d86b7714a1012014ce3533aa24dc88704fe14a6.1221271644.git.jbarnold@mit.edu> 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko The architecture-independent core of Ksplice. Ksplice makes it possible to apply certain kinds of patches to the kernel without having to reboot. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> Tested-by: Waseem Daher <wdaher@mit.edu> --- MAINTAINERS | 10 + arch/Kconfig | 14 + include/linux/ksplice.h | 202 +++++ kernel/Makefile | 2 + kernel/ksplice.c | 1997 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 2225 insertions(+), 0 deletions(-) create mode 100644 include/linux/ksplice.h create mode 100644 kernel/ksplice.c diff --git a/MAINTAINERS b/MAINTAINERS index 186be3b..bdc6dd7 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2476,6 +2476,16 @@ W: http://miguelojeda.es/auxdisplay.htm W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained +KSPLICE: +P: Jeffrey Brian Arnold +M: jbarnold@mit.edu +P: Anders Kaseorg +M: andersk@mit.edu +P: Tim Abbott +M: tabbott@mit.edu +W: http://web.mit.edu/ksplice +S: Maintained + LAPB module L: linux-x25@vger.kernel.org S: Orphan diff --git a/arch/Kconfig b/arch/Kconfig index 364c6da..9d5a843 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -27,6 +27,17 @@ config KPROBES for kernel debugging, non-intrusive instrumentation and testing. If in doubt, say "N". +config KSPLICE + bool "Ksplice rebootless kernel updates" + depends on KALLSYMS_ALL && MODULE_UNLOAD && SYSFS && \ + FUNCTION_DATA_SECTIONS + depends on HAVE_KSPLICE + help + Say Y here if you want to be able to apply certain kinds of + patches to your running kernel, without rebooting. + + If unsure, say N. + config HAVE_EFFICIENT_UNALIGNED_ACCESS def_bool n help @@ -56,6 +67,9 @@ config HAVE_IOREMAP_PROT config HAVE_KPROBES def_bool n +config HAVE_KSPLICE + def_bool n + config HAVE_KRETPROBES def_bool n diff --git a/include/linux/ksplice.h b/include/linux/ksplice.h new file mode 100644 index 0000000..d60f14f --- /dev/null +++ b/include/linux/ksplice.h @@ -0,0 +1,202 @@ +#include <linux/types.h> + +/** + * struct ksplice_symbol - Ksplice's analogue of an ELF symbol + * @name: The ELF name of the symbol + * @label: A unique Ksplice name for the symbol + **/ +struct ksplice_symbol { + const char *name; + const char *label; +}; + +/** + * struct ksplice_reloc - Ksplice's analogue of an ELF relocation + * @blank_addr: The address of the relocation's storage unit + * @blank_offset: The offset (from the start of the section) of the + * relocation's storage unit + * @symbol: The ksplice_symbol associated with this relocation + * @pcrel: Is the relocation PC relative? + * @addend: The ELF addend of the relocation + * @size: The size, in bytes, of the item to be relocated + * @dst_mask: Bitmask for which parts of the instruction or data are + * replaced with the relocated value + * (based on dst_mask from GNU BFD's reloc_howto_struct) + * @rightshift: The value the final relocation is shifted right by; + * used to drop unwanted data from the relocation + * (based on rightshift from GNU BFD's reloc_howto_struct) + * @signed_addend: Should the addend be interpreted as a signed value? + **/ +struct ksplice_reloc { + unsigned long blank_addr; + long blank_offset; + const struct ksplice_symbol *symbol; + int pcrel; + long addend; + int size; + long dst_mask; + unsigned int rightshift; + int signed_addend; +}; + +#if BITS_PER_LONG == 32 +#define KSPLICE_CANARY 0x77777777UL +#elif BITS_PER_LONG == 64 +#define KSPLICE_CANARY 0x7777777777777777UL +#endif /* BITS_PER_LONG */ + +/** + * struct ksplice_section - Ksplice's analogue of an ELF section + * @symbol: The ksplice_symbol associated with this section + * @size: The length, in bytes, of this section + * @address: The address of the section + * @flags: Specifies whether this section contains text, read-only + * data, or data + **/ +struct ksplice_section { + const struct ksplice_symbol *symbol; + unsigned long address; + unsigned long size; + unsigned int flags; +}; +#define KSPLICE_SECTION_TEXT 0x00000001 +#define KSPLICE_SECTION_RODATA 0x00000002 +#define KSPLICE_SECTION_DATA 0x00000004 + +#define MAX_TRAMPOLINE_SIZE 5 + +/** + * struct ksplice_trampoline - A trampoline Ksplice should insert + * @repladdr: The address of the replacement function + * @oldaddr: The address of the obsolete function + * @trampoline: The bytes of the trampoline itself + * @saved: The bytes of the original function which were + * overwritten by the trampoline + * @size: The size of the trampoline + **/ +struct ksplice_trampoline { + unsigned long repladdr; +/* private: */ + unsigned long oldaddr; + char trampoline[MAX_TRAMPOLINE_SIZE]; + char saved[MAX_TRAMPOLINE_SIZE]; + unsigned int size; +}; + +/** + * struct ksplice_patch - A function replacement that Ksplice should perform + * @label: The unique Ksplice name for the obsolete function + * @trampoline: A trampoline to insert over the obsolete function + * @reverse_trampoline: Used to simplify Ksplice internal bookkeeping + **/ +struct ksplice_patch { + const char *label; + struct ksplice_trampoline trampoline; +/* private: */ + struct ksplice_trampoline reverse_trampoline; +}; + +/** + * struct ksplice_export - A change to be made to the exported symbol table + * @name: The obsolete name of the exported symbol + * @new_name: The new name of the exported symbol + * @sym: The kernel_symbol being changed + * @saved_name: The pointer to the original name of the kernel_symbol + **/ +struct ksplice_export { + const char *name; + const char *new_name; +/* private: */ + struct kernel_symbol *sym; + const char *saved_name; +}; + +#ifdef __KERNEL__ +#include <linux/module.h> + +#undef _STR +#define _STR(x) #x +#undef STR +#define STR(x) _STR(x) +#define _PASTE(x, y) x##y +#define PASTE(x, y) _PASTE(x, y) +#define KSPLICE_UNIQ(s) PASTE(s##_, KSPLICE_MID) +#define KSPLICE_KID_UNIQ(s) PASTE(s##_, KSPLICE_KID) + +/** + * struct ksplice_module_list_entry - A record of a Ksplice pack's target + * @target: A module that is patched + * @primary: A Ksplice module that patches target + **/ +struct ksplice_module_list_entry { + struct module *target; + struct module *primary; +/* private: */ + struct list_head list; +}; + +/* List of all ksplice modules and the module they patch */ +extern struct list_head ksplice_module_list; + +/** + * struct ksplice_pack - Data for one module modified by a Ksplice update + * @name: The name of the primary module for the pack + * @kid: The Ksplice unique identifier for the pack + * @target_name: The name of the module modified by the pack + * @primary: The primary module associated with the pack + * @primary_relocs: The relocations for the primary module + * @primary_relocs_end: The end pointer for primary_relocs + * @primary_sections: The sections in the primary module + * @primary_sections_end: The end pointer for primary_sections array + * @helper_relocs: The relocations for the helper module + * @helper_relocs_end: The end pointer for helper_relocs array + * @helper_sections: The sections in the helper module + * @helper_sections_end: The end pointer for helper_sections array + * @patches: The function replacements in the pack + * @patches_end: The end pointer for patches array + * @exports: The exported symbol changes in the pack + * @exports_end: The end pointer for the exports array + * @update: The atomic update the pack is part of + * @target: The module modified by the pack + * @labelvals: The mapping between Ksplice symbol labels and + * their values + * @safety_records: The ranges of addresses that must not be on a + * kernel stack for the patch to apply safely + **/ +struct ksplice_pack { + const char *name; + const char *kid; + const char *target_name; + struct module *primary; + const struct ksplice_reloc *primary_relocs, *primary_relocs_end; + const struct ksplice_section *primary_sections, *primary_sections_end; + const struct ksplice_reloc *helper_relocs, *helper_relocs_end; + const struct ksplice_section *helper_sections, *helper_sections_end; + struct ksplice_patch *patches, *patches_end; + struct ksplice_export *exports, *exports_end; +/* private: */ + struct ksplice_module_list_entry module_list_entry; + struct update *update; + struct module *target; + struct list_head labelvals; + struct list_head safety_records; + struct list_head list; +}; + + +/** + * init_ksplice_pack() - Initializes a pack + * @pack: The pack to be initialized. All of the public fields of the + * pack and its associated data structures should be populated + * before this function is called. The values of the private + * fields will be ignored. + **/ +int init_ksplice_pack(struct ksplice_pack *pack); + +/** + * cleanup_ksplice_pack() - Cleans up a pack + * @pack: The pack to be cleaned up + */ +void cleanup_ksplice_pack(struct ksplice_pack *pack); + +#endif /* __KERNEL__ */ diff --git a/kernel/Makefile b/kernel/Makefile index 4e1d7df..26077f5 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -12,6 +12,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o CFLAGS_REMOVE_sched.o = -mno-spe +CFLAGS_ksplice.o += -Iarch/$(SRCARCH)/kernel ifdef CONFIG_FTRACE # Do not trace debug files and internal ftrace files @@ -68,6 +69,7 @@ obj-$(CONFIG_AUDIT) += audit.o auditfilter.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o obj-$(CONFIG_AUDIT_TREE) += audit_tree.o obj-$(CONFIG_KPROBES) += kprobes.o +obj-$(CONFIG_KSPLICE) += ksplice.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ diff --git a/kernel/ksplice.c b/kernel/ksplice.c new file mode 100644 index 0000000..b2e840a --- /dev/null +++ b/kernel/ksplice.c @@ -0,0 +1,1997 @@ +/* Copyright (C) 2007-2008 Jeffrey Brian Arnold <jbarnold@mit.edu> + * Copyright (C) 2008 Anders Kaseorg <andersk@mit.edu>, + * Tim Abbott <tabbott@mit.edu> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +#include <linux/module.h> +#include <linux/debugfs.h> +#include <linux/errno.h> +#include <linux/kallsyms.h> +#include <linux/kobject.h> +#include <linux/kthread.h> +#include <linux/pagemap.h> +#include <linux/sched.h> +#include <linux/stop_machine.h> +#include <linux/sysfs.h> +#include <linux/time.h> +#include <linux/uaccess.h> +#include <linux/vmalloc.h> +#include <linux/ksplice.h> + +enum stage { + STAGE_PREPARING, STAGE_APPLIED, STAGE_REVERSED +}; + +enum run_pre_mode { + RUN_PRE_INITIAL, RUN_PRE_DEBUG, RUN_PRE_FINAL +}; + +typedef int __bitwise__ abort_t; + +#define OK ((__force abort_t) 0) +#define NO_MATCH ((__force abort_t) 1) +#define CODE_BUSY ((__force abort_t) 2) +#define MODULE_BUSY ((__force abort_t) 3) +#define OUT_OF_MEMORY ((__force abort_t) 4) +#define FAILED_TO_FIND ((__force abort_t) 5) +#define ALREADY_REVERSED ((__force abort_t) 6) +#define MISSING_EXPORT ((__force abort_t) 7) +#define UNEXPECTED_RUNNING_TASK ((__force abort_t) 8) +#define UNEXPECTED ((__force abort_t) 9) + +struct update { + const char *kid; + const char *name; + struct kobject kobj; + enum stage stage; + abort_t abort_cause; + int debug; +#ifdef CONFIG_DEBUG_FS + struct debugfs_blob_wrapper debug_blob; + struct dentry *debugfs_dentry; +#else /* !CONFIG_DEBUG_FS */ + bool debug_continue_line; +#endif /* CONFIG_DEBUG_FS */ + struct list_head packs; + struct list_head conflicts; + struct list_head list; +}; + +struct conflict { + const char *process_name; + pid_t pid; + struct list_head stack; + struct list_head list; +}; + +struct conflict_addr { + unsigned long addr; + bool has_conflict; + const char *label; + struct list_head list; +}; + +struct labelval { + struct list_head list; + const char *label; + unsigned long val; + enum { NOVAL, TEMP, VAL } status; +}; + +struct safety_record { + struct list_head list; + const char *label; + unsigned long addr; + unsigned long size; + bool first_byte_safe; +}; + +struct candidate_val { + struct list_head list; + unsigned long val; +}; + +struct accumulate_struct { + struct ksplice_pack *pack; + const char *desired_name; + struct list_head *vals; +}; + +static LIST_HEAD(updates); +LIST_HEAD(ksplice_module_list); +EXPORT_SYMBOL_GPL(ksplice_module_list); +static struct kobject *ksplice_kobj; + +static struct kobj_type ksplice_ktype; + +static struct update *init_ksplice_update(const char *kid); +static void cleanup_ksplice_update(struct update *update); +static void add_to_update(struct ksplice_pack *pack, struct update *update); +static int ksplice_sysfs_init(struct update *update); + +/* Preparing the relocations and patches for application */ +static abort_t apply_update(struct update *update); +static abort_t prepare_pack(struct ksplice_pack *pack); +static abort_t finalize_pack(struct ksplice_pack *pack); +static abort_t finalize_exports(struct ksplice_pack *pack); +static abort_t finalize_patches(struct ksplice_pack *pack); +static abort_t add_dependency_on_address(struct ksplice_pack *pack, + unsigned long addr); +static abort_t apply_relocs(struct ksplice_pack *pack, + const struct ksplice_reloc *relocs, + const struct ksplice_reloc *relocs_end); +static abort_t apply_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r); +static abort_t read_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long *valp); +static abort_t write_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long sym_addr); +static void __attribute__((noreturn)) ksplice_deleted(void); + +/* run-pre matching */ +static abort_t match_pack_sections(struct ksplice_pack *pack, + bool consider_data_sections); +static abort_t find_section(struct ksplice_pack *pack, + const struct ksplice_section *sect); +static abort_t try_addr(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode); +static abort_t run_pre_cmp(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode); +static void print_bytes(struct ksplice_pack *pack, + const unsigned char *run, int runc, + const unsigned char *pre, int prec); +static abort_t lookup_reloc(struct ksplice_pack *pack, unsigned long addr, + const struct ksplice_reloc **relocp); +static abort_t handle_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long run_addr, enum run_pre_mode mode); + +/* Computing possible addresses for symbols */ +static abort_t lookup_symbol(struct ksplice_pack *pack, + const struct ksplice_symbol *ksym, + struct list_head *vals); +static abort_t lookup_symbol_kallsyms(struct ksplice_pack *pack, + const char *name, struct list_head *vals); +static int accumulate_matching_names(void *data, const char *sym_name, + struct module *sym_owner, + unsigned long sym_val); +static abort_t exported_symbol_lookup(const char *name, struct list_head *vals); +static abort_t new_export_lookup(struct update *update, + const char *name, struct list_head *vals); + +/* Atomic update insertion and removal */ +static abort_t apply_patches(struct update *update); +static abort_t reverse_patches(struct update *update); +static int __apply_patches(void *update); +static int __reverse_patches(void *update); +static abort_t check_each_task(struct update *update); +static abort_t check_task(struct update *update, + const struct task_struct *t, bool rerun); +static abort_t check_stack(struct update *update, struct conflict *conf, + const struct thread_info *tinfo, + const unsigned long *stack); +static abort_t check_address(struct update *update, + struct conflict *conf, unsigned long addr); +static abort_t check_record(struct conflict_addr *ca, + const struct safety_record *rec, + unsigned long addr); +static bool is_stop_machine(const struct task_struct *t); +static void cleanup_conflicts(struct update *update); +static void print_conflicts(struct update *update); +static void insert_trampoline(struct ksplice_trampoline *t); +static abort_t verify_trampoline(struct ksplice_pack *pack, + const struct ksplice_trampoline *t); +static void remove_trampoline(const struct ksplice_trampoline *t); + +static struct labelval *find_labelval(struct ksplice_pack *pack, + const char *label); +static abort_t create_labelval(struct ksplice_pack *pack, const char *label, + unsigned long val, int status); +static abort_t create_safety_record(struct ksplice_pack *pack, + const struct ksplice_section *sect, + struct list_head *record_list, + unsigned long run_addr, + unsigned long run_size); +static abort_t add_candidate_val(struct list_head *vals, unsigned long val); +static void prune_trampoline_vals(struct ksplice_pack *pack, + struct list_head *vals); +static void release_vals(struct list_head *vals); +static void set_temp_labelvals(struct ksplice_pack *pack, int status_val); + +static int contains_canary(struct ksplice_pack *pack, unsigned long blank_addr, + int size, long dst_mask); +static unsigned long follow_trampolines(struct ksplice_pack *pack, + unsigned long addr); +static bool patches_module(const struct module *a, const struct module *b); +static bool starts_with(const char *str, const char *prefix); +static bool singular(struct list_head *list); + +/* Debugging */ +static abort_t init_debug_buf(struct update *update); +static void clear_debug_buf(struct update *update); +static int __attribute__((format(printf, 2, 3))) +_ksdebug(struct update *update, const char *fmt, ...); +#define ksdebug(pack, fmt, ...) \ + _ksdebug(pack->update, fmt, ## __VA_ARGS__) + +/* Architecture-specific functions defined in arch/ARCH/kernel/ksplice-arch.c */ +static abort_t prepare_trampoline(struct ksplice_pack *pack, + struct ksplice_trampoline *t); +static abort_t trampoline_target(struct ksplice_pack *pack, unsigned long addr, + unsigned long *new_addr); +static abort_t handle_paravirt(struct ksplice_pack *pack, unsigned long pre, + unsigned long run, int *matched); +static bool valid_stack_ptr(const struct thread_info *tinfo, const void *p); + +#include "ksplice-arch.c" + +#define clear_list(head, type, member) \ + do { \ + struct list_head *_pos, *_n; \ + list_for_each_safe(_pos, _n, head) { \ + list_del(_pos); \ + kfree(list_entry(_pos, type, member)); \ + } \ + } while (0) + +int init_ksplice_pack(struct ksplice_pack *pack) +{ + struct update *update; + int ret = 0; + + INIT_LIST_HEAD(&pack->labelvals); + INIT_LIST_HEAD(&pack->safety_records); + + mutex_lock(&module_mutex); + if (strcmp(pack->target_name, "vmlinux") == 0) { + pack->target = NULL; + } else { + pack->target = find_module(pack->target_name); + if (pack->target == NULL || !module_is_live(pack->target)) { + ret = -ENODEV; + goto out; + } + ret = use_module(pack->primary, pack->target); + if (ret != 1) { + ret = -ENODEV; + goto out; + } + } + list_for_each_entry(update, &updates, list) { + if (strcmp(pack->kid, update->kid) == 0) { + if (update->stage != STAGE_PREPARING) { + ret = -EPERM; + goto out; + } + add_to_update(pack, update); + goto out; + } + } + update = init_ksplice_update(pack->kid); + if (update == NULL) { + ret = -ENOMEM; + goto out; + } + ret = ksplice_sysfs_init(update); + if (ret != 0) { + cleanup_ksplice_update(update); + goto out; + } + add_to_update(pack, update); +out: + mutex_unlock(&module_mutex); + return ret; +} +EXPORT_SYMBOL(init_ksplice_pack); + +void cleanup_ksplice_pack(struct ksplice_pack *pack) +{ + if (pack->update == NULL || pack->update->stage == STAGE_APPLIED) + return; + mutex_lock(&module_mutex); + list_del(&pack->list); + mutex_unlock(&module_mutex); + if (list_empty(&pack->update->packs)) + kobject_put(&pack->update->kobj); + pack->update = NULL; +} +EXPORT_SYMBOL_GPL(cleanup_ksplice_pack); + +static struct update *init_ksplice_update(const char *kid) +{ + struct update *update; + update = kcalloc(1, sizeof(struct update), GFP_KERNEL); + if (update == NULL) + return NULL; + update->name = kasprintf(GFP_KERNEL, "ksplice_%s", kid); + if (update->name == NULL) { + kfree(update); + return NULL; + } + update->kid = kstrdup(kid, GFP_KERNEL); + if (update->kid == NULL) { + kfree(update->name); + kfree(update); + return NULL; + } + INIT_LIST_HEAD(&update->packs); + if (init_debug_buf(update) != OK) { + kfree(update->kid); + kfree(update->name); + kfree(update); + return NULL; + } + list_add(&update->list, &updates); + update->stage = STAGE_PREPARING; + update->abort_cause = OK; + INIT_LIST_HEAD(&update->conflicts); + return update; +} + +static void cleanup_ksplice_update(struct update *update) +{ + mutex_lock(&module_mutex); + list_del(&update->list); + mutex_unlock(&module_mutex); + cleanup_conflicts(update); + clear_debug_buf(update); + kfree(update->kid); + kfree(update->name); + kfree(update); +} + +static void add_to_update(struct ksplice_pack *pack, struct update *update) +{ + pack->update = update; + list_add(&pack->list, &update->packs); + pack->module_list_entry.target = pack->target; + pack->module_list_entry.primary = pack->primary; +} + +static int ksplice_sysfs_init(struct update *update) +{ + int ret = 0; + memset(&update->kobj, 0, sizeof(update->kobj)); + ret = kobject_init_and_add(&update->kobj, &ksplice_ktype, + ksplice_kobj, "%s", update->kid); + if (ret != 0) + return ret; + kobject_uevent(&update->kobj, KOBJ_ADD); + return 0; +} + +static abort_t apply_update(struct update *update) +{ + struct ksplice_pack *pack; + abort_t ret; + + mutex_lock(&module_mutex); + + list_for_each_entry(pack, &update->packs, list) { + ret = prepare_pack(pack); + if (ret != OK) + goto out; + } + ret = apply_patches(update); +out: + list_for_each_entry(pack, &update->packs, list) { + clear_list(&pack->labelvals, struct labelval, list); + if (update->stage == STAGE_PREPARING) + clear_list(&pack->safety_records, struct safety_record, + list); + } + mutex_unlock(&module_mutex); + return ret; +} + + +static abort_t prepare_pack(struct ksplice_pack *pack) +{ + abort_t ret; + + ksdebug(pack, "Preparing and checking %s\n", pack->name); + ret = match_pack_sections(pack, false); + if (ret == NO_MATCH) { + /* It is possible that by using relocations from .data sections + we can successfully run-pre match the rest of the sections. + To avoid using any symbols obtained from .data sections + (which may be unreliable) in the post code, we first prepare + the post code and then try to run-pre match the remaining + sections with the help of .data sections. + */ + ksdebug(pack, "Continuing without some sections; we might " + "find them later.\n"); + ret = finalize_pack(pack); + if (ret != OK) { + ksdebug(pack, "Aborted. Unable to continue without " + "the unmatched sections.\n"); + return ret; + } + + ksdebug(pack, "run-pre: Considering .data sections to find the " + "unmatched sections\n"); + ret = match_pack_sections(pack, true); + if (ret != OK) + return ret; + + ksdebug(pack, "run-pre: Found all previously unmatched " + "sections\n"); + return OK; + } else if (ret != OK) { + return ret; + } + + return finalize_pack(pack); +} + +static abort_t finalize_pack(struct ksplice_pack *pack) +{ + abort_t ret; + ret = apply_relocs(pack, pack->primary_relocs, + pack->primary_relocs_end); + if (ret != OK) + return ret; + + ret = finalize_patches(pack); + if (ret != OK) + return ret; + + ret = finalize_exports(pack); + if (ret != OK) + return ret; + + return OK; +} + +static abort_t finalize_exports(struct ksplice_pack *pack) +{ + struct ksplice_export *exp; + struct module *m; + const struct kernel_symbol *sym; + + for (exp = pack->exports; exp < pack->exports_end; exp++) { + sym = find_symbol(exp->name, &m, NULL, true, false); + if (sym == NULL) { + ksdebug(pack, "Could not find kernel_symbol struct for " + "%s\n", exp->name); + return MISSING_EXPORT; + } + + /* Cast away const since we are planning to mutate the + * kernel_symbol structure. */ + exp->sym = (struct kernel_symbol *)sym; + exp->saved_name = exp->sym->name; + if (m != pack->primary && use_module(pack->primary, m) != 1) { + ksdebug(pack, "Aborted. Could not add dependency on " + "symbol %s from module %s.\n", sym->name, + m->name); + return UNEXPECTED; + } + } + return OK; +} + +static abort_t finalize_patches(struct ksplice_pack *pack) +{ + struct ksplice_patch *p; + struct safety_record *rec; + abort_t ret; + + for (p = pack->patches; p < pack->patches_end; p++) { + struct labelval *lv = find_labelval(pack, p->label); + bool found = false; + if (lv == NULL) { + ksdebug(pack, "Failed to find %s for oldaddr\n", + p->label); + return FAILED_TO_FIND; + } + p->trampoline.oldaddr = lv->val; + + list_for_each_entry(rec, &pack->safety_records, list) { + if (strcmp(rec->label, p->label) == 0 && + follow_trampolines(pack, p->trampoline.oldaddr) + == rec->addr) { + found = true; + break; + } + } + if (!found) { + ksdebug(pack, "No safety record for patch %s\n", + p->label); + return NO_MATCH; + } + if (rec->size < p->trampoline.size) { + ksdebug(pack, "Symbol %s is too short for trampoline\n", + p->label); + return UNEXPECTED; + } + + if (p->trampoline.repladdr == 0) + p->trampoline.repladdr = (unsigned long)ksplice_deleted; + else + rec->first_byte_safe = true; + + ret = prepare_trampoline(pack, &p->trampoline); + if (ret != OK) + return ret; + + if (p->trampoline.oldaddr != rec->addr) { + /* If there's already a trampoline at oldaddr, prepare + a reverse trampoline to install there */ + p->reverse_trampoline.oldaddr = rec->addr; + p->reverse_trampoline.repladdr = p->trampoline.oldaddr; + ret = prepare_trampoline(pack, &p->reverse_trampoline); + if (ret != OK) + return ret; + } else { + p->reverse_trampoline.size = 0; + } + + ret = add_dependency_on_address(pack, p->trampoline.oldaddr); + if (ret != OK) + return ret; + } + return OK; +} + +static abort_t add_dependency_on_address(struct ksplice_pack *pack, + unsigned long addr) +{ + struct module *m = + __module_text_address(follow_trampolines(pack, addr)); + if (m == NULL || m == pack->primary) + return OK; + if (use_module(pack->primary, m) != 1) + return MODULE_BUSY; + return OK; +} + +static abort_t apply_relocs(struct ksplice_pack *pack, + const struct ksplice_reloc *relocs, + const struct ksplice_reloc *relocs_end) +{ + const struct ksplice_reloc *r; + for (r = relocs; r < relocs_end; r++) { + abort_t ret = apply_reloc(pack, r); + if (ret != OK) + return ret; + } + return OK; +} + +static abort_t apply_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r) +{ + abort_t ret; + int canary_ret; + unsigned long sym_addr; + LIST_HEAD(vals); + + canary_ret = contains_canary(pack, r->blank_addr, r->size, r->dst_mask); + if (canary_ret < 0) + return UNEXPECTED; + if (canary_ret == 0) { + ksdebug(pack, "reloc: skipped %s:%lx (altinstr)\n", + r->symbol->label, r->blank_offset); + return OK; + } + + ret = lookup_symbol(pack, r->symbol, &vals); + if (ret != OK) { + release_vals(&vals); + return ret; + } + if (!singular(&vals)) { + release_vals(&vals); + ksdebug(pack, "Failed to find %s for reloc\n", + r->symbol->label); + return FAILED_TO_FIND; + } + sym_addr = list_entry(vals.next, struct candidate_val, list)->val; + release_vals(&vals); + + ret = write_reloc_value(pack, r, r->blank_addr, + r->pcrel ? sym_addr - r->blank_addr : sym_addr); + if (ret != OK) + return ret; + + ksdebug(pack, "reloc: %s:%lx", r->symbol->label, r->blank_offset); + ksdebug(pack, "(S=%lx A=%lx ", sym_addr, r->addend); + switch (r->size) { + case 1: + ksdebug(pack, "aft=%02x)\n", *(uint8_t *)r->blank_addr); + break; + case 2: + ksdebug(pack, "aft=%04x)\n", *(uint16_t *)r->blank_addr); + break; + case 4: + ksdebug(pack, "aft=%08x)\n", *(uint32_t *)r->blank_addr); + break; +#if BITS_PER_LONG >= 64 + case 8: + ksdebug(pack, "aft=%016llx)\n", *(uint64_t *)r->blank_addr); + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + /* Create labelvals so that we can verify our choices in the second + round of run-pre matching that considers data sections. */ + ret = create_labelval(pack, r->symbol->label, sym_addr, VAL); + if (ret != OK) + return ret; + return add_dependency_on_address(pack, sym_addr); +} + +static abort_t read_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long *valp) +{ + unsigned char bytes[sizeof(long)]; + unsigned long val; + + if (probe_kernel_read(bytes, (void *)addr, r->size) == -EFAULT) + return NO_MATCH; + + switch (r->size) { + case 1: + val = *(uint8_t *)bytes; + break; + case 2: + val = *(uint16_t *)bytes; + break; + case 4: + val = *(uint32_t *)bytes; + break; +#if BITS_PER_LONG >= 64 + case 8: + val = *(uint64_t *)bytes; + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + + val &= r->dst_mask; + if (r->signed_addend) + val |= -(val & (r->dst_mask & ~(r->dst_mask >> 1))); + val <<= r->rightshift; + val -= r->addend; + *valp = val; + return OK; +} + +static abort_t write_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long sym_addr) +{ + unsigned long val = sym_addr + r->addend; + val >>= r->rightshift; + switch (r->size) { + case 1: + *(uint8_t *)addr = + (*(uint8_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; + case 2: + *(uint16_t *)addr = + (*(uint16_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; + case 4: + *(uint32_t *)addr = + (*(uint32_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; +#if BITS_PER_LONG >= 64 + case 8: + *(uint64_t *)addr = + (*(uint64_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + + if (read_reloc_value(pack, r, addr, &val) != OK || val != sym_addr) { + ksdebug(pack, "Aborted. Relocation overflow.\n"); + return UNEXPECTED; + } + + return OK; +} + +static void __attribute__((noreturn)) ksplice_deleted(void) +{ + printk(KERN_CRIT "Called a kernel function deleted by Ksplice!\n"); + BUG(); +} + +static abort_t match_pack_sections(struct ksplice_pack *pack, + bool consider_data_sections) +{ + const struct ksplice_section *sect; + abort_t ret; + char *finished; + int i, remaining = 0; + bool progress; + + finished = kcalloc(pack->helper_sections_end - pack->helper_sections, + sizeof(*finished), GFP_KERNEL); + if (finished == NULL) + return OUT_OF_MEMORY; + for (sect = pack->helper_sections; sect < pack->helper_sections_end; + sect++) { + if ((sect->flags & KSPLICE_SECTION_DATA) == 0) + remaining++; + } + + while (remaining > 0) { + progress = false; + for (sect = pack->helper_sections; + sect < pack->helper_sections_end; sect++) { + i = sect - pack->helper_sections; + if (finished[i]) + continue; + if (!consider_data_sections && + (sect->flags & KSPLICE_SECTION_DATA) != 0) + continue; + ret = find_section(pack, sect); + if (ret == OK) { + finished[i] = 1; + if ((sect->flags & KSPLICE_SECTION_DATA) == 0) + remaining--; + progress = true; + } else if (ret != NO_MATCH) { + kfree(finished); + return ret; + } + } + + if (progress) + continue; + + for (sect = pack->helper_sections; + sect < pack->helper_sections_end; sect++) { + i = sect - pack->helper_sections; + if (finished[i] == 0) + ksdebug(pack, "run-pre: could not match " + "section %s\n", sect->symbol->label); + } + ksdebug(pack, "Aborted. run-pre: could not match some " + "sections.\n"); + kfree(finished); + return NO_MATCH; + } + kfree(finished); + return OK; +} + +static abort_t find_section(struct ksplice_pack *pack, + const struct ksplice_section *sect) +{ + int i; + abort_t ret; + unsigned long run_addr; + LIST_HEAD(vals); + struct candidate_val *v, *n; + + ret = lookup_symbol(pack, sect->symbol, &vals); + if (ret != OK) { + release_vals(&vals); + return ret; + } + + ksdebug(pack, "run-pre: starting sect search for %s\n", + sect->symbol->label); + + list_for_each_entry_safe(v, n, &vals, list) { + run_addr = v->val; + + yield(); + ret = try_addr(pack, sect, run_addr, NULL, RUN_PRE_INITIAL); + if (ret == NO_MATCH) { + list_del(&v->list); + kfree(v); + } else if (ret != OK) { + release_vals(&vals); + return ret; + } + } + + prune_trampoline_vals(pack, &vals); + if (singular(&vals)) { + LIST_HEAD(safety_records); + run_addr = list_entry(vals.next, struct candidate_val, + list)->val; + ret = try_addr(pack, sect, run_addr, &safety_records, + RUN_PRE_FINAL); + release_vals(&vals); + if (ret != OK) { + clear_list(&safety_records, struct safety_record, list); + ksdebug(pack, "run-pre: Final run failed for sect " + "%s:\n", sect->symbol->label); + } else { + list_splice(&safety_records, &pack->safety_records); + } + return ret; + } else if (!list_empty(&vals)) { + struct candidate_val *val; + ksdebug(pack, "run-pre: multiple candidates for sect %s:\n", + sect->symbol->label); + i = 0; + list_for_each_entry(val, &vals, list) { + i++; + ksdebug(pack, "%lx\n", val->val); + if (i > 5) { + ksdebug(pack, "...\n"); + break; + } + } + release_vals(&vals); + return NO_MATCH; + } + release_vals(&vals); + return NO_MATCH; +} + +static abort_t try_addr(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode) +{ + abort_t ret; + const struct module *run_module; + + if ((sect->flags & KSPLICE_SECTION_RODATA) != 0 || + (sect->flags & KSPLICE_SECTION_DATA) != 0) + run_module = __module_data_address(run_addr); + else + run_module = __module_text_address(run_addr); + if (!patches_module(run_module, pack->target)) { + ksdebug(pack, "run-pre: ignoring address %lx in other module " + "%s for sect %s\n", run_addr, run_module == NULL ? + "vmlinux" : run_module->name, sect->symbol->label); + return NO_MATCH; + } + + ret = create_labelval(pack, sect->symbol->label, run_addr, TEMP); + if (ret != OK) + return ret; + + ret = run_pre_cmp(pack, sect, run_addr, safety_records, mode); + if (ret == NO_MATCH && mode != RUN_PRE_FINAL) { + set_temp_labelvals(pack, NOVAL); + ksdebug(pack, "run-pre: %s sect %s does not match (r_a=%lx " + "p_a=%lx s=%lx)\n", + (sect->flags & KSPLICE_SECTION_RODATA) != 0 ? "data" : + "text", sect->symbol->label, run_addr, sect->address, + sect->size); + ksdebug(pack, "run-pre: "); + if (pack->update->debug >= 1) { + ret = run_pre_cmp(pack, sect, run_addr, safety_records, + RUN_PRE_DEBUG); + set_temp_labelvals(pack, NOVAL); + } + ksdebug(pack, "\n"); + return ret; + } else if (ret != OK) { + set_temp_labelvals(pack, NOVAL); + return ret; + } + + if (mode != RUN_PRE_FINAL) { + set_temp_labelvals(pack, NOVAL); + ksdebug(pack, "run-pre: candidate for sect %s=%lx\n", + sect->symbol->label, run_addr); + return OK; + } + + set_temp_labelvals(pack, VAL); + ksdebug(pack, "run-pre: found sect %s=%lx\n", sect->symbol->label, + run_addr); + return OK; +} + +static abort_t run_pre_cmp(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode) +{ + int matched = 0; + abort_t ret; + const struct ksplice_reloc *r; + const unsigned char *pre, *run, *pre_start, *run_start; + unsigned char runval; + + if ((sect->flags & KSPLICE_SECTION_TEXT) != 0) + run_addr = follow_trampolines(pack, run_addr); + + pre_start = (const unsigned char *)sect->address; + run_start = (const unsigned char *)run_addr; + + pre = pre_start; + run = run_start; + while (pre < pre_start + sect->size) { + unsigned long offset = pre - pre_start; + ret = lookup_reloc(pack, (unsigned long)pre, &r); + if (ret == OK) { + ret = handle_reloc(pack, r, (unsigned long)run, mode); + if (ret != OK) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "reloc in sect does not " + "match after %lx/%lx bytes\n", + offset, sect->size); + return ret; + } + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, r->size, pre, r->size); + pre += r->size; + run += r->size; + continue; + } else if (ret != NO_MATCH) { + return ret; + } + + if ((sect->flags & KSPLICE_SECTION_TEXT) != 0) { + ret = handle_paravirt(pack, (unsigned long)pre, + (unsigned long)run, &matched); + if (ret != OK) + return ret; + if (matched != 0) { + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, matched, pre, + matched); + pre += matched; + run += matched; + continue; + } + } + + if (probe_kernel_read(&runval, (void *)run, 1) == -EFAULT) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "sect unmapped after %lx/%lx " + "bytes\n", offset, sect->size); + return NO_MATCH; + } + + if (runval != *pre && + (sect->flags & KSPLICE_SECTION_DATA) == 0) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "sect does not match after " + "%lx/%lx bytes\n", offset, sect->size); + if (mode == RUN_PRE_DEBUG) { + print_bytes(pack, run, 1, pre, 1); + ksdebug(pack, "[p_o=%lx] ! ", offset); + print_bytes(pack, run + 1, 2, pre + 1, 2); + } + return NO_MATCH; + } + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, 1, pre, 1); + pre++; + run++; + } + return create_safety_record(pack, sect, safety_records, run_addr, + run - run_start); +} + +static void print_bytes(struct ksplice_pack *pack, + const unsigned char *run, int runc, + const unsigned char *pre, int prec) +{ + int o; + int matched = min(runc, prec); + for (o = 0; o < matched; o++) { + if (run[o] == pre[o]) + ksdebug(pack, "%02x ", run[o]); + else + ksdebug(pack, "%02x/%02x ", run[o], pre[o]); + } + for (o = matched; o < runc; o++) + ksdebug(pack, "%02x/ ", run[o]); + for (o = matched; o < prec; o++) + ksdebug(pack, "/%02x ", pre[o]); +} + +static abort_t lookup_reloc(struct ksplice_pack *pack, unsigned long addr, + const struct ksplice_reloc **relocp) +{ + const struct ksplice_reloc *r; + int canary_ret; + for (r = pack->helper_relocs; r < pack->helper_relocs_end; r++) { + if (addr >= r->blank_addr && addr < r->blank_addr + r->size) { + canary_ret = contains_canary(pack, r->blank_addr, + r->size, r->dst_mask); + if (canary_ret < 0) + return UNEXPECTED; + if (canary_ret == 0) { + ksdebug(pack, "reloc: skipped %s:%lx " + "(altinstr)\n", r->symbol->label, + r->blank_offset); + return NO_MATCH; + } + if (addr != r->blank_addr) { + ksdebug(pack, "Invalid nonzero relocation " + "offset\n"); + return UNEXPECTED; + } + *relocp = r; + return OK; + } + } + return NO_MATCH; +} + +static abort_t handle_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long run_addr, enum run_pre_mode mode) +{ + unsigned long val; + abort_t ret; + + ret = read_reloc_value(pack, r, run_addr, &val); + if (ret != OK) + return ret; + if (r->pcrel) + val += run_addr; + + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "run-pre: reloc at r_a=%lx p_a=%lx to %s+%lx: " + "found %s = %lx\n", run_addr, r->blank_addr, + r->symbol->label, r->addend, r->symbol->label, val); + + if (starts_with(r->symbol->label, ".rodata.str")) + return OK; + + if (contains_canary(pack, run_addr, r->size, r->dst_mask) != 0) { + ksdebug(pack, "Aborted. Unexpected canary in run code at %lx" + "\n", run_addr); + return UNEXPECTED; + } + + ret = create_labelval(pack, r->symbol->label, val, TEMP); + if (ret == NO_MATCH && mode == RUN_PRE_INITIAL) { + struct labelval *lv = find_labelval(pack, r->symbol->label); + ksdebug(pack, "run-pre: reloc at r_a=%lx p_a=%lx: labelval %s " + "= %lx(%d) does not match expected %lx\n", run_addr, + r->blank_addr, r->symbol->label, lv->val, lv->status, + val); + } + return ret; +} + +static abort_t lookup_symbol(struct ksplice_pack *pack, + const struct ksplice_symbol *ksym, + struct list_head *vals) +{ + abort_t ret; + struct labelval *lv; + + lv = find_labelval(pack, ksym->label); + if (lv != NULL) { + release_vals(vals); + ksdebug(pack, "using detected sym %s=%lx\n", ksym->label, + lv->val); + return add_candidate_val(vals, lv->val); + } + + if (starts_with(ksym->label, ".rodata.str")) + return OK; + + if (strcmp(ksym->label, "cleanup_module") == 0 && pack->target != NULL + && pack->target->exit != NULL) { + ret = add_candidate_val(vals, + (unsigned long)pack->target->exit); + if (ret != OK) + return ret; + } + + ret = exported_symbol_lookup(ksym->name, vals); + if (ret != OK) + return ret; + + ret = new_export_lookup(pack->update, ksym->name, vals); + if (ret != OK) + return ret; + + ret = lookup_symbol_kallsyms(pack, ksym->name, vals); + if (ret != OK) + return ret; + + return OK; +} + +static abort_t lookup_symbol_kallsyms(struct ksplice_pack *pack, + const char *name, struct list_head *vals) +{ + struct accumulate_struct acc = { pack, name, vals }; + return (__force abort_t) + kallsyms_on_each_symbol(accumulate_matching_names, &acc); +} + +static int accumulate_matching_names(void *data, const char *sym_name, + struct module *sym_owner, + unsigned long sym_val) +{ + struct accumulate_struct *acc = data; + if (strcmp(sym_name, acc->desired_name) == 0 && + patches_module(sym_owner, acc->pack->target) && + sym_owner != acc->pack->primary) + return (__force int)add_candidate_val(acc->vals, sym_val); + return (__force int)OK; +} + +static abort_t exported_symbol_lookup(const char *name, struct list_head *vals) +{ + const struct kernel_symbol *sym; + sym = find_symbol(name, NULL, NULL, true, false); + if (sym == NULL) + return OK; + return add_candidate_val(vals, sym->value); +} + +static abort_t new_export_lookup(struct update *update, + const char *name, struct list_head *vals) +{ + struct ksplice_pack *pack; + struct ksplice_export *exp; + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) { + if (strcmp(exp->new_name, name) == 0 && + exp->sym != NULL && + contains_canary(pack, + (unsigned long)&exp->sym->value, + sizeof(unsigned long), -1) == 0) + return add_candidate_val(vals, exp->sym->value); + } + } + return OK; +} + +static abort_t apply_patches(struct update *update) +{ + int i; + abort_t ret; + struct ksplice_pack *pack; + const struct ksplice_section *sect; + + list_for_each_entry(pack, &update->packs, list) { + for (sect = pack->primary_sections; + sect < pack->primary_sections_end; sect++) { + ret = create_safety_record(pack, sect, + &pack->safety_records, + sect->address, sect->size); + if (ret != OK) + return ret; + } + } + + for (i = 0; i < 5; i++) { + cleanup_conflicts(update); + ret = (__force abort_t)stop_machine(__apply_patches, update, + NULL); + if (ret != CODE_BUSY) + break; + set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(msecs_to_jiffies(1000)); + } + + if (ret == CODE_BUSY) { + print_conflicts(update); + _ksdebug(update, "Aborted %s. stack check: to-be-replaced " + "code is busy.\n", update->kid); + } else if (ret == ALREADY_REVERSED) { + _ksdebug(update, "Aborted %s. Ksplice update %s is already " + "reversed.\n", update->kid, update->kid); + } + + if (ret != OK) + return ret; + + _ksdebug(update, "Atomic patch insertion for %s complete\n", + update->kid); + return OK; +} + +static abort_t reverse_patches(struct update *update) +{ + int i; + abort_t ret; + struct ksplice_pack *pack; + + clear_debug_buf(update); + ret = init_debug_buf(update); + if (ret != OK) + return ret; + + _ksdebug(update, "Preparing to reverse %s\n", update->kid); + + for (i = 0; i < 5; i++) { + cleanup_conflicts(update); + clear_list(&update->conflicts, struct conflict, list); + ret = (__force abort_t)stop_machine(__reverse_patches, update, + NULL); + if (ret != CODE_BUSY) + break; + set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(msecs_to_jiffies(1000)); + } + list_for_each_entry(pack, &update->packs, list) + clear_list(&pack->safety_records, struct safety_record, list); + + if (ret == CODE_BUSY) { + print_conflicts(update); + _ksdebug(update, "Aborted %s. stack check: to-be-reversed " + "code is busy.\n", update->kid); + } else if (ret == MODULE_BUSY) { + _ksdebug(update, "Update %s is in use by another module\n", + update->kid); + } + + if (ret != OK) + return ret; + + _ksdebug(update, "Atomic patch removal for %s complete\n", update->kid); + return OK; +} + +static int __apply_patches(void *updateptr) +{ + struct update *update = updateptr; + struct ksplice_pack *pack; + struct ksplice_patch *p; + struct ksplice_export *exp; + abort_t ret; + + if (update->stage == STAGE_APPLIED) + return (__force int)OK; + + if (update->stage != STAGE_PREPARING) + return (__force int)UNEXPECTED; + + ret = check_each_task(update); + if (ret != OK) + return (__force int)ret; + + list_for_each_entry(pack, &update->packs, list) { + if (try_module_get(pack->primary) != 1) { + struct ksplice_pack *pack1; + list_for_each_entry(pack1, &update->packs, list) { + if (pack1 == pack) + break; + module_put(pack1->primary); + } + return (__force int)UNEXPECTED; + } + } + + update->stage = STAGE_APPLIED; + + list_for_each_entry(pack, &update->packs, list) + list_add(&pack->module_list_entry.list, &ksplice_module_list); + + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) + exp->sym->name = exp->new_name; + } + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + insert_trampoline(&p->trampoline); + insert_trampoline(&p->reverse_trampoline); + } + } + return (__force int)OK; +} + +static int __reverse_patches(void *updateptr) +{ + struct update *update = updateptr; + struct ksplice_pack *pack; + const struct ksplice_patch *p; + struct ksplice_export *exp; + abort_t ret; + + if (update->stage != STAGE_APPLIED) + return (__force int)OK; + + list_for_each_entry(pack, &update->packs, list) { + if (module_refcount(pack->primary) != 1) + return (__force int)MODULE_BUSY; + } + + ret = check_each_task(update); + if (ret != OK) + return (__force int)ret; + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + ret = verify_trampoline(pack, &p->trampoline); + if (ret != OK) + return (__force int)ret; + ret = verify_trampoline(pack, &p->reverse_trampoline); + if (ret != OK) + return (__force int)ret; + } + } + + update->stage = STAGE_REVERSED; + + list_for_each_entry(pack, &update->packs, list) + module_put(pack->primary); + + list_for_each_entry(pack, &update->packs, list) + list_del(&pack->module_list_entry.list); + + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) + exp->sym->name = exp->saved_name; + } + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + remove_trampoline(&p->trampoline); + remove_trampoline(&p->reverse_trampoline); + } + } + return (__force int)OK; +} + +static abort_t check_each_task(struct update *update) +{ + const struct task_struct *g, *p; + abort_t status = OK, ret; + do_each_thread(g, p) { + /* do_each_thread is a double loop! */ + ret = check_task(update, p, false); + if (ret != OK) { + check_task(update, p, true); + status = ret; + } + if (ret != OK && ret != CODE_BUSY) + return ret; + } while_each_thread(g, p); + return status; +} + +static abort_t check_task(struct update *update, + const struct task_struct *t, bool rerun) +{ + abort_t status, ret; + struct conflict *conf = NULL; + + if (rerun) { + conf = kmalloc(sizeof(*conf), GFP_ATOMIC); + if (conf == NULL) + return OUT_OF_MEMORY; + conf->process_name = kstrdup(t->comm, GFP_ATOMIC); + if (conf->process_name == NULL) { + kfree(conf); + return OUT_OF_MEMORY; + } + conf->pid = t->pid; + INIT_LIST_HEAD(&conf->stack); + list_add(&conf->list, &update->conflicts); + } + + status = check_address(update, conf, KSPLICE_IP(t)); + if (t == current) { + ret = check_stack(update, conf, task_thread_info(t), + (unsigned long *)__builtin_frame_address(0)); + if (status == OK) + status = ret; + } else if (!task_curr(t)) { + ret = check_stack(update, conf, task_thread_info(t), + (unsigned long *)KSPLICE_SP(t)); + if (status == OK) + status = ret; + } else if (!is_stop_machine(t)) { + status = UNEXPECTED_RUNNING_TASK; + } + return status; +} + +static abort_t check_stack(struct update *update, struct conflict *conf, + const struct thread_info *tinfo, + const unsigned long *stack) +{ + abort_t status = OK, ret; + unsigned long addr; + + while (valid_stack_ptr(tinfo, stack)) { + addr = *stack++; + ret = check_address(update, conf, addr); + if (ret != OK) + status = ret; + } + return status; +} + +static abort_t check_address(struct update *update, + struct conflict *conf, unsigned long addr) +{ + abort_t status = OK, ret; + const struct safety_record *rec; + struct ksplice_pack *pack; + struct conflict_addr *ca = NULL; + + if (conf != NULL) { + ca = kmalloc(sizeof(*ca), GFP_ATOMIC); + if (ca == NULL) + return OUT_OF_MEMORY; + ca->addr = addr; + ca->has_conflict = false; + ca->label = NULL; + list_add(&ca->list, &conf->stack); + } + + list_for_each_entry(pack, &update->packs, list) { + list_for_each_entry(rec, &pack->safety_records, list) { + ret = check_record(ca, rec, addr); + if (ret != OK) + status = ret; + } + } + return status; +} + +static abort_t check_record(struct conflict_addr *ca, + const struct safety_record *rec, unsigned long addr) +{ + if ((addr > rec->addr && addr < rec->addr + rec->size) || + (addr == rec->addr && !rec->first_byte_safe)) { + if (ca != NULL) { + ca->label = rec->label; + ca->has_conflict = true; + } + return CODE_BUSY; + } + return OK; +} + +static bool is_stop_machine(const struct task_struct *t) +{ + const char *num; + if (!starts_with(t->comm, "kstop")) + return false; + num = t->comm + strlen("kstop"); + return num[strspn(num, "0123456789")] == '\0'; +} + +static void cleanup_conflicts(struct update *update) +{ + struct conflict *conf; + list_for_each_entry(conf, &update->conflicts, list) { + clear_list(&conf->stack, struct conflict_addr, list); + kfree(conf->process_name); + } + clear_list(&update->conflicts, struct conflict, list); +} + +static void print_conflicts(struct update *update) +{ + const struct conflict *conf; + const struct conflict_addr *ca; + list_for_each_entry(conf, &update->conflicts, list) { + _ksdebug(update, "stack check: pid %d (%s):", conf->pid, + conf->process_name); + list_for_each_entry(ca, &conf->stack, list) { + _ksdebug(update, " %lx", ca->addr); + if (ca->has_conflict) + _ksdebug(update, " [<-CONFLICT]"); + } + _ksdebug(update, "\n"); + } +} + +static void insert_trampoline(struct ksplice_trampoline *t) +{ + mm_segment_t old_fs = get_fs(); + set_fs(KERNEL_DS); + memcpy((void *)t->saved, (void *)t->oldaddr, t->size); + memcpy((void *)t->oldaddr, (void *)t->trampoline, t->size); + flush_icache_range(t->oldaddr, t->oldaddr + t->size); + set_fs(old_fs); +} + +static abort_t verify_trampoline(struct ksplice_pack *pack, + const struct ksplice_trampoline *t) +{ + if (memcmp((void *)t->oldaddr, (void *)t->trampoline, t->size) != 0) { + ksdebug(pack, "Aborted. Trampoline at %lx has been " + "overwritten.\n", t->oldaddr); + return CODE_BUSY; + } + return OK; +} + +static void remove_trampoline(const struct ksplice_trampoline *t) +{ + mm_segment_t old_fs = get_fs(); + set_fs(KERNEL_DS); + memcpy((void *)t->oldaddr, (void *)t->saved, t->size); + flush_icache_range(t->oldaddr, t->oldaddr + t->size); + set_fs(old_fs); +} + +static struct labelval *find_labelval(struct ksplice_pack *pack, + const char *label) +{ + struct labelval *lv; + list_for_each_entry(lv, &pack->labelvals, list) { + if (strcmp(lv->label, label) == 0) + return lv; + } + return NULL; +} + +static abort_t create_labelval(struct ksplice_pack *pack, const char *label, + unsigned long val, int status) +{ + struct labelval *lv = find_labelval(pack, label); + if (lv != NULL) + return lv->val == val ? OK : NO_MATCH; + + lv = kmalloc(sizeof(*lv), GFP_KERNEL); + if (lv == NULL) + return OUT_OF_MEMORY; + lv->label = label; + lv->val = val; + lv->status = status; + list_add(&lv->list, &pack->labelvals); + return OK; +} + +static abort_t create_safety_record(struct ksplice_pack *pack, + const struct ksplice_section *sect, + struct list_head *record_list, + unsigned long run_addr, + unsigned long run_size) +{ + struct safety_record *rec; + struct ksplice_patch *p; + + if (record_list == NULL) + return OK; + + for (p = pack->patches; p < pack->patches_end; p++) { + if (strcmp(sect->symbol->label, p->label) == 0) + break; + } + if (p >= pack->patches_end) + return OK; + + if ((sect->flags & KSPLICE_SECTION_TEXT) == 0 && + p->trampoline.repladdr != 0) { + ksdebug(pack, "Error: ksplice_patch %s is matched to a " + "non-deleted non-text section!\n", sect->symbol->label); + return UNEXPECTED; + } + + rec = kmalloc(sizeof(*rec), GFP_KERNEL); + if (rec == NULL) + return OUT_OF_MEMORY; + rec->addr = run_addr; + rec->size = run_size; + rec->label = sect->symbol->label; + rec->first_byte_safe = false; + + list_add(&rec->list, record_list); + return OK; +} + +static abort_t add_candidate_val(struct list_head *vals, unsigned long val) +{ + struct candidate_val *tmp, *new; + + list_for_each_entry(tmp, vals, list) { + if (tmp->val == val) + return OK; + } + new = kmalloc(sizeof(*new), GFP_KERNEL); + if (new == NULL) + return OUT_OF_MEMORY; + new->val = val; + list_add(&new->list, vals); + return OK; +} + +/* If there are only two candidates and their addresses are related by + a trampoline, then we have successfully found a function patched by a + previous update. We remove the endpoint of the trampoline from the vals + list, so that this update uses the patched function's original address. */ +static void prune_trampoline_vals(struct ksplice_pack *pack, + struct list_head *vals) +{ + struct candidate_val *val1, *val2; + + if (list_empty(vals) || singular(vals)) + return; + if (vals->next->next->next != vals) + return; + + val1 = list_entry(vals->next, struct candidate_val, list); + val2 = list_entry(vals->next->next, struct candidate_val, list); + + if (val1->val == follow_trampolines(pack, val2->val)) { + list_del(&val1->list); + kfree(val1); + return; + } + if (val2->val == follow_trampolines(pack, val1->val)) { + list_del(&val2->list); + kfree(val2); + return; + } +} + +static void release_vals(struct list_head *vals) +{ + clear_list(vals, struct candidate_val, list); +} + +static void set_temp_labelvals(struct ksplice_pack *pack, int status) +{ + struct labelval *lv, *n; + list_for_each_entry_safe(lv, n, &pack->labelvals, list) { + if (lv->status == TEMP) { + if (status == NOVAL) { + list_del(&lv->list); + kfree(lv); + } else { + lv->status = status; + } + } + } +} + +static int contains_canary(struct ksplice_pack *pack, unsigned long blank_addr, + int size, long dst_mask) +{ + switch (size) { + case 1: + return (*(uint8_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); + case 2: + return (*(uint16_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); + case 4: + return (*(uint32_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); +#if BITS_PER_LONG >= 64 + case 8: + return (*(uint64_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return -1; + } +} + +static unsigned long follow_trampolines(struct ksplice_pack *pack, + unsigned long addr) +{ + unsigned long new_addr; + struct module *m; + + if (trampoline_target(pack, addr, &new_addr) != OK) + return addr; + + /* Confirm that it is a jump into a ksplice module */ + m = __module_text_address(new_addr); + if (m != NULL && m != pack->target && starts_with(m->name, "ksplice")) { + ksdebug(pack, "Following trampoline %lx %lx(%s)\n", addr, + new_addr, m->name); + return new_addr; + } + return addr; +} + +/* Does module a patch module b? */ +static bool patches_module(const struct module *a, const struct module *b) +{ + struct ksplice_module_list_entry *entry; + if (a == b) + return true; + list_for_each_entry(entry, &ksplice_module_list, list) { + if (entry->target == b && entry->primary == a) + return true; + } + return false; +} + +static bool starts_with(const char *str, const char *prefix) +{ + return strncmp(str, prefix, strlen(prefix)) == 0; +} + +static bool singular(struct list_head *list) +{ + return !list_empty(list) && list->next->next == list; +} + +#ifdef CONFIG_DEBUG_FS + +static abort_t init_debug_buf(struct update *update) +{ + update->debug_blob.size = 0; + update->debug_blob.data = NULL; + update->debugfs_dentry = + debugfs_create_blob(update->name, S_IFREG | S_IRUSR, NULL, + &update->debug_blob); + if (update->debugfs_dentry == NULL) + return OUT_OF_MEMORY; + return OK; +} + +static void clear_debug_buf(struct update *update) +{ + if (update->debugfs_dentry == NULL) + return; + debugfs_remove(update->debugfs_dentry); + update->debugfs_dentry = NULL; + update->debug_blob.size = 0; + vfree(update->debug_blob.data); + update->debug_blob.data = NULL; +} + +static int _ksdebug(struct update *update, const char *fmt, ...) +{ + va_list args; + unsigned long size, old_size, new_size; + + if (update->debug == 0) + return 0; + + /* size includes the trailing '\0' */ + va_start(args, fmt); + size = 1 + vsnprintf(update->debug_blob.data, 0, fmt, args); + va_end(args); + old_size = update->debug_blob.size == 0 ? 0 : + max(PAGE_SIZE, roundup_pow_of_two(update->debug_blob.size)); + new_size = update->debug_blob.size + size == 0 ? 0 : + max(PAGE_SIZE, roundup_pow_of_two(update->debug_blob.size + size)); + if (new_size > old_size) { + char *buf = vmalloc(new_size); + if (buf == NULL) + return -ENOMEM; + memcpy(buf, update->debug_blob.data, update->debug_blob.size); + vfree(update->debug_blob.data); + update->debug_blob.data = buf; + } + va_start(args, fmt); + update->debug_blob.size += vsnprintf(update->debug_blob.data + + update->debug_blob.size, + size, fmt, args); + va_end(args); + return 0; +} +#else /* CONFIG_DEBUG_FS */ +static abort_t init_debug_buf(struct update *update) +{ + return OK; +} + +static void clear_debug_buf(struct update *update) +{ + return; +} + +static int _ksdebug(struct update *update, const char *fmt, ...) +{ + va_list args; + + if (update->debug == 0) + return 0; + + if (!update->debug_continue_line) + printk(KERN_DEBUG "ksplice: "); + + va_start(args, fmt); + vprintk(fmt, args); + va_end(args); + + update->debug_continue_line = + fmt[0] == '\0' || fmt[strlen(fmt) - 1] != '\n'; + return 0; +} +#endif /* CONFIG_DEBUG_FS */ + +struct ksplice_attribute { + struct attribute attr; + ssize_t (*show)(struct update *update, char *buf); + ssize_t (*store)(struct update *update, const char *buf, size_t len); +}; + +static ssize_t ksplice_attr_show(struct kobject *kobj, struct attribute *attr, + char *buf) +{ + struct ksplice_attribute *attribute = + container_of(attr, struct ksplice_attribute, attr); + struct update *update = container_of(kobj, struct update, kobj); + if (attribute->show == NULL) + return -EIO; + return attribute->show(update, buf); +} + +static ssize_t ksplice_attr_store(struct kobject *kobj, struct attribute *attr, + const char *buf, size_t len) +{ + struct ksplice_attribute *attribute = + container_of(attr, struct ksplice_attribute, attr); + struct update *update = container_of(kobj, struct update, kobj); + if (attribute->store == NULL) + return -EIO; + return attribute->store(update, buf, len); +} + +static struct sysfs_ops ksplice_sysfs_ops = { + .show = ksplice_attr_show, + .store = ksplice_attr_store, +}; + +static void ksplice_release(struct kobject *kobj) +{ + struct update *update; + update = container_of(kobj, struct update, kobj); + cleanup_ksplice_update(update); +} + +static ssize_t stage_show(struct update *update, char *buf) +{ + switch (update->stage) { + case STAGE_PREPARING: + return snprintf(buf, PAGE_SIZE, "preparing\n"); + case STAGE_APPLIED: + return snprintf(buf, PAGE_SIZE, "applied\n"); + case STAGE_REVERSED: + return snprintf(buf, PAGE_SIZE, "reversed\n"); + } + return 0; +} + +static ssize_t abort_cause_show(struct update *update, char *buf) +{ + switch (update->abort_cause) { + case OK: + return snprintf(buf, PAGE_SIZE, "ok\n"); + case NO_MATCH: + return snprintf(buf, PAGE_SIZE, "no_match\n"); + case CODE_BUSY: + return snprintf(buf, PAGE_SIZE, "code_busy\n"); + case MODULE_BUSY: + return snprintf(buf, PAGE_SIZE, "module_busy\n"); + case OUT_OF_MEMORY: + return snprintf(buf, PAGE_SIZE, "out_of_memory\n"); + case FAILED_TO_FIND: + return snprintf(buf, PAGE_SIZE, "failed_to_find\n"); + case ALREADY_REVERSED: + return snprintf(buf, PAGE_SIZE, "already_reversed\n"); + case MISSING_EXPORT: + return snprintf(buf, PAGE_SIZE, "missing_export\n"); + case UNEXPECTED_RUNNING_TASK: + return snprintf(buf, PAGE_SIZE, "unexpected_running_task\n"); + case UNEXPECTED: + return snprintf(buf, PAGE_SIZE, "unexpected\n"); + } + return 0; +} + +static ssize_t conflict_show(struct update *update, char *buf) +{ + const struct conflict *conf; + const struct conflict_addr *ca; + int used = 0; + list_for_each_entry(conf, &update->conflicts, list) { + used += snprintf(buf + used, PAGE_SIZE - used, "%s %d", + conf->process_name, conf->pid); + list_for_each_entry(ca, &conf->stack, list) { + if (!ca->has_conflict) + continue; + used += snprintf(buf + used, PAGE_SIZE - used, " %s", + ca->label); + } + used += snprintf(buf + used, PAGE_SIZE - used, "\n"); + } + return used; +} + +static ssize_t stage_store(struct update *update, const char *buf, size_t len) +{ + if ((strncmp(buf, "applied", len) == 0 || + strncmp(buf, "applied\n", len) == 0) && + update->stage == STAGE_PREPARING) + update->abort_cause = apply_update(update); + else if ((strncmp(buf, "reversed", len) == 0 || + strncmp(buf, "reversed\n", len) == 0) && + update->stage == STAGE_APPLIED) + update->abort_cause = reverse_patches(update); + if (update->abort_cause == OK) + printk(KERN_INFO "ksplice: Update %s %s successfully\n", + update->kid, + update->stage == STAGE_APPLIED ? "applied" : "reversed"); + return len; +} + +static ssize_t debug_show(struct update *update, char *buf) +{ + return snprintf(buf, PAGE_SIZE, "%d\n", update->debug); +} + +static ssize_t debug_store(struct update *update, const char *buf, size_t len) +{ + unsigned long l; + int ret = strict_strtoul(buf, 10, &l); + if (ret != 0) + return ret; + update->debug = l; + return len; +} + +static struct ksplice_attribute stage_attribute = + __ATTR(stage, 0600, stage_show, stage_store); +static struct ksplice_attribute abort_cause_attribute = + __ATTR(abort_cause, 0400, abort_cause_show, NULL); +static struct ksplice_attribute debug_attribute = + __ATTR(debug, 0600, debug_show, debug_store); +static struct ksplice_attribute conflict_attribute = + __ATTR(conflicts, 0400, conflict_show, NULL); + +static struct attribute *ksplice_attrs[] = { + &stage_attribute.attr, + &abort_cause_attribute.attr, + &debug_attribute.attr, + &conflict_attribute.attr, + NULL +}; + +static struct kobj_type ksplice_ktype = { + .sysfs_ops = &ksplice_sysfs_ops, + .release = ksplice_release, + .default_attrs = ksplice_attrs, +}; + +static int init_ksplice(void) +{ + ksplice_kobj = kobject_create_and_add("ksplice", kernel_kobj); + if (ksplice_kobj == NULL) + return -ENOMEM; + return 0; +} + +static void cleanup_ksplice(void) +{ + kobject_put(ksplice_kobj); +} + +module_init(init_ksplice); +module_exit(cleanup_ksplice); + +MODULE_AUTHOR("Jeffrey Brian Arnold <jbarnold@mit.edu>"); +MODULE_DESCRIPTION("Ksplice rebootless update system"); +MODULE_LICENSE("GPL v2"); -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
[parent not found: <8d86b7714a1012014ce3533aa24dc88704fe14a6.1221271644.git.jbarnold@mit.edu>]
* [RFC PATCH 5/9] Ksplice core (architecture independent) [not found] ` <8d86b7714a1012014ce3533aa24dc88704fe14a6.1221271644.git.jbarnold@mit.edu> @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 6/9] Ksplice: Documentation Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 6/9] Ksplice documentation Jeffrey Brian Arnold 1 sibling, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko The architecture-independent core of Ksplice. Ksplice makes it possible to apply certain kinds of patches to the kernel without having to reboot. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> Tested-by: Waseem Daher <wdaher@mit.edu> --- MAINTAINERS | 10 + arch/Kconfig | 14 + include/linux/ksplice.h | 202 +++++ kernel/Makefile | 2 + kernel/ksplice.c | 1997 +++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 2225 insertions(+), 0 deletions(-) create mode 100644 include/linux/ksplice.h create mode 100644 kernel/ksplice.c diff --git a/MAINTAINERS b/MAINTAINERS index 186be3b..bdc6dd7 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2476,6 +2476,16 @@ W: http://miguelojeda.es/auxdisplay.htm W: http://jair.lab.fi.uva.es/~migojed/auxdisplay.htm S: Maintained +KSPLICE: +P: Jeffrey Brian Arnold +M: jbarnold@mit.edu +P: Anders Kaseorg +M: andersk@mit.edu +P: Tim Abbott +M: tabbott@mit.edu +W: http://web.mit.edu/ksplice +S: Maintained + LAPB module L: linux-x25@vger.kernel.org S: Orphan diff --git a/arch/Kconfig b/arch/Kconfig index 364c6da..9d5a843 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -27,6 +27,17 @@ config KPROBES for kernel debugging, non-intrusive instrumentation and testing. If in doubt, say "N". +config KSPLICE + bool "Ksplice rebootless kernel updates" + depends on KALLSYMS_ALL && MODULE_UNLOAD && SYSFS && \ + FUNCTION_DATA_SECTIONS + depends on HAVE_KSPLICE + help + Say Y here if you want to be able to apply certain kinds of + patches to your running kernel, without rebooting. + + If unsure, say N. + config HAVE_EFFICIENT_UNALIGNED_ACCESS def_bool n help @@ -56,6 +67,9 @@ config HAVE_IOREMAP_PROT config HAVE_KPROBES def_bool n +config HAVE_KSPLICE + def_bool n + config HAVE_KRETPROBES def_bool n diff --git a/include/linux/ksplice.h b/include/linux/ksplice.h new file mode 100644 index 0000000..d60f14f --- /dev/null +++ b/include/linux/ksplice.h @@ -0,0 +1,202 @@ +#include <linux/types.h> + +/** + * struct ksplice_symbol - Ksplice's analogue of an ELF symbol + * @name: The ELF name of the symbol + * @label: A unique Ksplice name for the symbol + **/ +struct ksplice_symbol { + const char *name; + const char *label; +}; + +/** + * struct ksplice_reloc - Ksplice's analogue of an ELF relocation + * @blank_addr: The address of the relocation's storage unit + * @blank_offset: The offset (from the start of the section) of the + * relocation's storage unit + * @symbol: The ksplice_symbol associated with this relocation + * @pcrel: Is the relocation PC relative? + * @addend: The ELF addend of the relocation + * @size: The size, in bytes, of the item to be relocated + * @dst_mask: Bitmask for which parts of the instruction or data are + * replaced with the relocated value + * (based on dst_mask from GNU BFD's reloc_howto_struct) + * @rightshift: The value the final relocation is shifted right by; + * used to drop unwanted data from the relocation + * (based on rightshift from GNU BFD's reloc_howto_struct) + * @signed_addend: Should the addend be interpreted as a signed value? + **/ +struct ksplice_reloc { + unsigned long blank_addr; + long blank_offset; + const struct ksplice_symbol *symbol; + int pcrel; + long addend; + int size; + long dst_mask; + unsigned int rightshift; + int signed_addend; +}; + +#if BITS_PER_LONG == 32 +#define KSPLICE_CANARY 0x77777777UL +#elif BITS_PER_LONG == 64 +#define KSPLICE_CANARY 0x7777777777777777UL +#endif /* BITS_PER_LONG */ + +/** + * struct ksplice_section - Ksplice's analogue of an ELF section + * @symbol: The ksplice_symbol associated with this section + * @size: The length, in bytes, of this section + * @address: The address of the section + * @flags: Specifies whether this section contains text, read-only + * data, or data + **/ +struct ksplice_section { + const struct ksplice_symbol *symbol; + unsigned long address; + unsigned long size; + unsigned int flags; +}; +#define KSPLICE_SECTION_TEXT 0x00000001 +#define KSPLICE_SECTION_RODATA 0x00000002 +#define KSPLICE_SECTION_DATA 0x00000004 + +#define MAX_TRAMPOLINE_SIZE 5 + +/** + * struct ksplice_trampoline - A trampoline Ksplice should insert + * @repladdr: The address of the replacement function + * @oldaddr: The address of the obsolete function + * @trampoline: The bytes of the trampoline itself + * @saved: The bytes of the original function which were + * overwritten by the trampoline + * @size: The size of the trampoline + **/ +struct ksplice_trampoline { + unsigned long repladdr; +/* private: */ + unsigned long oldaddr; + char trampoline[MAX_TRAMPOLINE_SIZE]; + char saved[MAX_TRAMPOLINE_SIZE]; + unsigned int size; +}; + +/** + * struct ksplice_patch - A function replacement that Ksplice should perform + * @label: The unique Ksplice name for the obsolete function + * @trampoline: A trampoline to insert over the obsolete function + * @reverse_trampoline: Used to simplify Ksplice internal bookkeeping + **/ +struct ksplice_patch { + const char *label; + struct ksplice_trampoline trampoline; +/* private: */ + struct ksplice_trampoline reverse_trampoline; +}; + +/** + * struct ksplice_export - A change to be made to the exported symbol table + * @name: The obsolete name of the exported symbol + * @new_name: The new name of the exported symbol + * @sym: The kernel_symbol being changed + * @saved_name: The pointer to the original name of the kernel_symbol + **/ +struct ksplice_export { + const char *name; + const char *new_name; +/* private: */ + struct kernel_symbol *sym; + const char *saved_name; +}; + +#ifdef __KERNEL__ +#include <linux/module.h> + +#undef _STR +#define _STR(x) #x +#undef STR +#define STR(x) _STR(x) +#define _PASTE(x, y) x##y +#define PASTE(x, y) _PASTE(x, y) +#define KSPLICE_UNIQ(s) PASTE(s##_, KSPLICE_MID) +#define KSPLICE_KID_UNIQ(s) PASTE(s##_, KSPLICE_KID) + +/** + * struct ksplice_module_list_entry - A record of a Ksplice pack's target + * @target: A module that is patched + * @primary: A Ksplice module that patches target + **/ +struct ksplice_module_list_entry { + struct module *target; + struct module *primary; +/* private: */ + struct list_head list; +}; + +/* List of all ksplice modules and the module they patch */ +extern struct list_head ksplice_module_list; + +/** + * struct ksplice_pack - Data for one module modified by a Ksplice update + * @name: The name of the primary module for the pack + * @kid: The Ksplice unique identifier for the pack + * @target_name: The name of the module modified by the pack + * @primary: The primary module associated with the pack + * @primary_relocs: The relocations for the primary module + * @primary_relocs_end: The end pointer for primary_relocs + * @primary_sections: The sections in the primary module + * @primary_sections_end: The end pointer for primary_sections array + * @helper_relocs: The relocations for the helper module + * @helper_relocs_end: The end pointer for helper_relocs array + * @helper_sections: The sections in the helper module + * @helper_sections_end: The end pointer for helper_sections array + * @patches: The function replacements in the pack + * @patches_end: The end pointer for patches array + * @exports: The exported symbol changes in the pack + * @exports_end: The end pointer for the exports array + * @update: The atomic update the pack is part of + * @target: The module modified by the pack + * @labelvals: The mapping between Ksplice symbol labels and + * their values + * @safety_records: The ranges of addresses that must not be on a + * kernel stack for the patch to apply safely + **/ +struct ksplice_pack { + const char *name; + const char *kid; + const char *target_name; + struct module *primary; + const struct ksplice_reloc *primary_relocs, *primary_relocs_end; + const struct ksplice_section *primary_sections, *primary_sections_end; + const struct ksplice_reloc *helper_relocs, *helper_relocs_end; + const struct ksplice_section *helper_sections, *helper_sections_end; + struct ksplice_patch *patches, *patches_end; + struct ksplice_export *exports, *exports_end; +/* private: */ + struct ksplice_module_list_entry module_list_entry; + struct update *update; + struct module *target; + struct list_head labelvals; + struct list_head safety_records; + struct list_head list; +}; + + +/** + * init_ksplice_pack() - Initializes a pack + * @pack: The pack to be initialized. All of the public fields of the + * pack and its associated data structures should be populated + * before this function is called. The values of the private + * fields will be ignored. + **/ +int init_ksplice_pack(struct ksplice_pack *pack); + +/** + * cleanup_ksplice_pack() - Cleans up a pack + * @pack: The pack to be cleaned up + */ +void cleanup_ksplice_pack(struct ksplice_pack *pack); + +#endif /* __KERNEL__ */ diff --git a/kernel/Makefile b/kernel/Makefile index 4e1d7df..26077f5 100644 --- a/kernel/Makefile +++ b/kernel/Makefile @@ -12,6 +12,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o CFLAGS_REMOVE_sched.o = -mno-spe +CFLAGS_ksplice.o += -Iarch/$(SRCARCH)/kernel ifdef CONFIG_FTRACE # Do not trace debug files and internal ftrace files @@ -68,6 +69,7 @@ obj-$(CONFIG_AUDIT) += audit.o auditfilter.o obj-$(CONFIG_AUDITSYSCALL) += auditsc.o obj-$(CONFIG_AUDIT_TREE) += audit_tree.o obj-$(CONFIG_KPROBES) += kprobes.o +obj-$(CONFIG_KSPLICE) += ksplice.o obj-$(CONFIG_KGDB) += kgdb.o obj-$(CONFIG_DETECT_SOFTLOCKUP) += softlockup.o obj-$(CONFIG_GENERIC_HARDIRQS) += irq/ diff --git a/kernel/ksplice.c b/kernel/ksplice.c new file mode 100644 index 0000000..b2e840a --- /dev/null +++ b/kernel/ksplice.c @@ -0,0 +1,1997 @@ +/* Copyright (C) 2007-2008 Jeffrey Brian Arnold <jbarnold@mit.edu> + * Copyright (C) 2008 Anders Kaseorg <andersk@mit.edu>, + * Tim Abbott <tabbott@mit.edu> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +#include <linux/module.h> +#include <linux/debugfs.h> +#include <linux/errno.h> +#include <linux/kallsyms.h> +#include <linux/kobject.h> +#include <linux/kthread.h> +#include <linux/pagemap.h> +#include <linux/sched.h> +#include <linux/stop_machine.h> +#include <linux/sysfs.h> +#include <linux/time.h> +#include <linux/uaccess.h> +#include <linux/vmalloc.h> +#include <linux/ksplice.h> + +enum stage { + STAGE_PREPARING, STAGE_APPLIED, STAGE_REVERSED +}; + +enum run_pre_mode { + RUN_PRE_INITIAL, RUN_PRE_DEBUG, RUN_PRE_FINAL +}; + +typedef int __bitwise__ abort_t; + +#define OK ((__force abort_t) 0) +#define NO_MATCH ((__force abort_t) 1) +#define CODE_BUSY ((__force abort_t) 2) +#define MODULE_BUSY ((__force abort_t) 3) +#define OUT_OF_MEMORY ((__force abort_t) 4) +#define FAILED_TO_FIND ((__force abort_t) 5) +#define ALREADY_REVERSED ((__force abort_t) 6) +#define MISSING_EXPORT ((__force abort_t) 7) +#define UNEXPECTED_RUNNING_TASK ((__force abort_t) 8) +#define UNEXPECTED ((__force abort_t) 9) + +struct update { + const char *kid; + const char *name; + struct kobject kobj; + enum stage stage; + abort_t abort_cause; + int debug; +#ifdef CONFIG_DEBUG_FS + struct debugfs_blob_wrapper debug_blob; + struct dentry *debugfs_dentry; +#else /* !CONFIG_DEBUG_FS */ + bool debug_continue_line; +#endif /* CONFIG_DEBUG_FS */ + struct list_head packs; + struct list_head conflicts; + struct list_head list; +}; + +struct conflict { + const char *process_name; + pid_t pid; + struct list_head stack; + struct list_head list; +}; + +struct conflict_addr { + unsigned long addr; + bool has_conflict; + const char *label; + struct list_head list; +}; + +struct labelval { + struct list_head list; + const char *label; + unsigned long val; + enum { NOVAL, TEMP, VAL } status; +}; + +struct safety_record { + struct list_head list; + const char *label; + unsigned long addr; + unsigned long size; + bool first_byte_safe; +}; + +struct candidate_val { + struct list_head list; + unsigned long val; +}; + +struct accumulate_struct { + struct ksplice_pack *pack; + const char *desired_name; + struct list_head *vals; +}; + +static LIST_HEAD(updates); +LIST_HEAD(ksplice_module_list); +EXPORT_SYMBOL_GPL(ksplice_module_list); +static struct kobject *ksplice_kobj; + +static struct kobj_type ksplice_ktype; + +static struct update *init_ksplice_update(const char *kid); +static void cleanup_ksplice_update(struct update *update); +static void add_to_update(struct ksplice_pack *pack, struct update *update); +static int ksplice_sysfs_init(struct update *update); + +/* Preparing the relocations and patches for application */ +static abort_t apply_update(struct update *update); +static abort_t prepare_pack(struct ksplice_pack *pack); +static abort_t finalize_pack(struct ksplice_pack *pack); +static abort_t finalize_exports(struct ksplice_pack *pack); +static abort_t finalize_patches(struct ksplice_pack *pack); +static abort_t add_dependency_on_address(struct ksplice_pack *pack, + unsigned long addr); +static abort_t apply_relocs(struct ksplice_pack *pack, + const struct ksplice_reloc *relocs, + const struct ksplice_reloc *relocs_end); +static abort_t apply_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r); +static abort_t read_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long *valp); +static abort_t write_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long sym_addr); +static void __attribute__((noreturn)) ksplice_deleted(void); + +/* run-pre matching */ +static abort_t match_pack_sections(struct ksplice_pack *pack, + bool consider_data_sections); +static abort_t find_section(struct ksplice_pack *pack, + const struct ksplice_section *sect); +static abort_t try_addr(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode); +static abort_t run_pre_cmp(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode); +static void print_bytes(struct ksplice_pack *pack, + const unsigned char *run, int runc, + const unsigned char *pre, int prec); +static abort_t lookup_reloc(struct ksplice_pack *pack, unsigned long addr, + const struct ksplice_reloc **relocp); +static abort_t handle_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long run_addr, enum run_pre_mode mode); + +/* Computing possible addresses for symbols */ +static abort_t lookup_symbol(struct ksplice_pack *pack, + const struct ksplice_symbol *ksym, + struct list_head *vals); +static abort_t lookup_symbol_kallsyms(struct ksplice_pack *pack, + const char *name, struct list_head *vals); +static int accumulate_matching_names(void *data, const char *sym_name, + struct module *sym_owner, + unsigned long sym_val); +static abort_t exported_symbol_lookup(const char *name, struct list_head *vals); +static abort_t new_export_lookup(struct update *update, + const char *name, struct list_head *vals); + +/* Atomic update insertion and removal */ +static abort_t apply_patches(struct update *update); +static abort_t reverse_patches(struct update *update); +static int __apply_patches(void *update); +static int __reverse_patches(void *update); +static abort_t check_each_task(struct update *update); +static abort_t check_task(struct update *update, + const struct task_struct *t, bool rerun); +static abort_t check_stack(struct update *update, struct conflict *conf, + const struct thread_info *tinfo, + const unsigned long *stack); +static abort_t check_address(struct update *update, + struct conflict *conf, unsigned long addr); +static abort_t check_record(struct conflict_addr *ca, + const struct safety_record *rec, + unsigned long addr); +static bool is_stop_machine(const struct task_struct *t); +static void cleanup_conflicts(struct update *update); +static void print_conflicts(struct update *update); +static void insert_trampoline(struct ksplice_trampoline *t); +static abort_t verify_trampoline(struct ksplice_pack *pack, + const struct ksplice_trampoline *t); +static void remove_trampoline(const struct ksplice_trampoline *t); + +static struct labelval *find_labelval(struct ksplice_pack *pack, + const char *label); +static abort_t create_labelval(struct ksplice_pack *pack, const char *label, + unsigned long val, int status); +static abort_t create_safety_record(struct ksplice_pack *pack, + const struct ksplice_section *sect, + struct list_head *record_list, + unsigned long run_addr, + unsigned long run_size); +static abort_t add_candidate_val(struct list_head *vals, unsigned long val); +static void prune_trampoline_vals(struct ksplice_pack *pack, + struct list_head *vals); +static void release_vals(struct list_head *vals); +static void set_temp_labelvals(struct ksplice_pack *pack, int status_val); + +static int contains_canary(struct ksplice_pack *pack, unsigned long blank_addr, + int size, long dst_mask); +static unsigned long follow_trampolines(struct ksplice_pack *pack, + unsigned long addr); +static bool patches_module(const struct module *a, const struct module *b); +static bool starts_with(const char *str, const char *prefix); +static bool singular(struct list_head *list); + +/* Debugging */ +static abort_t init_debug_buf(struct update *update); +static void clear_debug_buf(struct update *update); +static int __attribute__((format(printf, 2, 3))) +_ksdebug(struct update *update, const char *fmt, ...); +#define ksdebug(pack, fmt, ...) \ + _ksdebug(pack->update, fmt, ## __VA_ARGS__) + +/* Architecture-specific functions defined in arch/ARCH/kernel/ksplice-arch.c */ +static abort_t prepare_trampoline(struct ksplice_pack *pack, + struct ksplice_trampoline *t); +static abort_t trampoline_target(struct ksplice_pack *pack, unsigned long addr, + unsigned long *new_addr); +static abort_t handle_paravirt(struct ksplice_pack *pack, unsigned long pre, + unsigned long run, int *matched); +static bool valid_stack_ptr(const struct thread_info *tinfo, const void *p); + +#include "ksplice-arch.c" + +#define clear_list(head, type, member) \ + do { \ + struct list_head *_pos, *_n; \ + list_for_each_safe(_pos, _n, head) { \ + list_del(_pos); \ + kfree(list_entry(_pos, type, member)); \ + } \ + } while (0) + +int init_ksplice_pack(struct ksplice_pack *pack) +{ + struct update *update; + int ret = 0; + + INIT_LIST_HEAD(&pack->labelvals); + INIT_LIST_HEAD(&pack->safety_records); + + mutex_lock(&module_mutex); + if (strcmp(pack->target_name, "vmlinux") == 0) { + pack->target = NULL; + } else { + pack->target = find_module(pack->target_name); + if (pack->target == NULL || !module_is_live(pack->target)) { + ret = -ENODEV; + goto out; + } + ret = use_module(pack->primary, pack->target); + if (ret != 1) { + ret = -ENODEV; + goto out; + } + } + list_for_each_entry(update, &updates, list) { + if (strcmp(pack->kid, update->kid) == 0) { + if (update->stage != STAGE_PREPARING) { + ret = -EPERM; + goto out; + } + add_to_update(pack, update); + goto out; + } + } + update = init_ksplice_update(pack->kid); + if (update == NULL) { + ret = -ENOMEM; + goto out; + } + ret = ksplice_sysfs_init(update); + if (ret != 0) { + cleanup_ksplice_update(update); + goto out; + } + add_to_update(pack, update); +out: + mutex_unlock(&module_mutex); + return ret; +} +EXPORT_SYMBOL(init_ksplice_pack); + +void cleanup_ksplice_pack(struct ksplice_pack *pack) +{ + if (pack->update == NULL || pack->update->stage == STAGE_APPLIED) + return; + mutex_lock(&module_mutex); + list_del(&pack->list); + mutex_unlock(&module_mutex); + if (list_empty(&pack->update->packs)) + kobject_put(&pack->update->kobj); + pack->update = NULL; +} +EXPORT_SYMBOL_GPL(cleanup_ksplice_pack); + +static struct update *init_ksplice_update(const char *kid) +{ + struct update *update; + update = kcalloc(1, sizeof(struct update), GFP_KERNEL); + if (update == NULL) + return NULL; + update->name = kasprintf(GFP_KERNEL, "ksplice_%s", kid); + if (update->name == NULL) { + kfree(update); + return NULL; + } + update->kid = kstrdup(kid, GFP_KERNEL); + if (update->kid == NULL) { + kfree(update->name); + kfree(update); + return NULL; + } + INIT_LIST_HEAD(&update->packs); + if (init_debug_buf(update) != OK) { + kfree(update->kid); + kfree(update->name); + kfree(update); + return NULL; + } + list_add(&update->list, &updates); + update->stage = STAGE_PREPARING; + update->abort_cause = OK; + INIT_LIST_HEAD(&update->conflicts); + return update; +} + +static void cleanup_ksplice_update(struct update *update) +{ + mutex_lock(&module_mutex); + list_del(&update->list); + mutex_unlock(&module_mutex); + cleanup_conflicts(update); + clear_debug_buf(update); + kfree(update->kid); + kfree(update->name); + kfree(update); +} + +static void add_to_update(struct ksplice_pack *pack, struct update *update) +{ + pack->update = update; + list_add(&pack->list, &update->packs); + pack->module_list_entry.target = pack->target; + pack->module_list_entry.primary = pack->primary; +} + +static int ksplice_sysfs_init(struct update *update) +{ + int ret = 0; + memset(&update->kobj, 0, sizeof(update->kobj)); + ret = kobject_init_and_add(&update->kobj, &ksplice_ktype, + ksplice_kobj, "%s", update->kid); + if (ret != 0) + return ret; + kobject_uevent(&update->kobj, KOBJ_ADD); + return 0; +} + +static abort_t apply_update(struct update *update) +{ + struct ksplice_pack *pack; + abort_t ret; + + mutex_lock(&module_mutex); + + list_for_each_entry(pack, &update->packs, list) { + ret = prepare_pack(pack); + if (ret != OK) + goto out; + } + ret = apply_patches(update); +out: + list_for_each_entry(pack, &update->packs, list) { + clear_list(&pack->labelvals, struct labelval, list); + if (update->stage == STAGE_PREPARING) + clear_list(&pack->safety_records, struct safety_record, + list); + } + mutex_unlock(&module_mutex); + return ret; +} + + +static abort_t prepare_pack(struct ksplice_pack *pack) +{ + abort_t ret; + + ksdebug(pack, "Preparing and checking %s\n", pack->name); + ret = match_pack_sections(pack, false); + if (ret == NO_MATCH) { + /* It is possible that by using relocations from .data sections + we can successfully run-pre match the rest of the sections. + To avoid using any symbols obtained from .data sections + (which may be unreliable) in the post code, we first prepare + the post code and then try to run-pre match the remaining + sections with the help of .data sections. + */ + ksdebug(pack, "Continuing without some sections; we might " + "find them later.\n"); + ret = finalize_pack(pack); + if (ret != OK) { + ksdebug(pack, "Aborted. Unable to continue without " + "the unmatched sections.\n"); + return ret; + } + + ksdebug(pack, "run-pre: Considering .data sections to find the " + "unmatched sections\n"); + ret = match_pack_sections(pack, true); + if (ret != OK) + return ret; + + ksdebug(pack, "run-pre: Found all previously unmatched " + "sections\n"); + return OK; + } else if (ret != OK) { + return ret; + } + + return finalize_pack(pack); +} + +static abort_t finalize_pack(struct ksplice_pack *pack) +{ + abort_t ret; + ret = apply_relocs(pack, pack->primary_relocs, + pack->primary_relocs_end); + if (ret != OK) + return ret; + + ret = finalize_patches(pack); + if (ret != OK) + return ret; + + ret = finalize_exports(pack); + if (ret != OK) + return ret; + + return OK; +} + +static abort_t finalize_exports(struct ksplice_pack *pack) +{ + struct ksplice_export *exp; + struct module *m; + const struct kernel_symbol *sym; + + for (exp = pack->exports; exp < pack->exports_end; exp++) { + sym = find_symbol(exp->name, &m, NULL, true, false); + if (sym == NULL) { + ksdebug(pack, "Could not find kernel_symbol struct for " + "%s\n", exp->name); + return MISSING_EXPORT; + } + + /* Cast away const since we are planning to mutate the + * kernel_symbol structure. */ + exp->sym = (struct kernel_symbol *)sym; + exp->saved_name = exp->sym->name; + if (m != pack->primary && use_module(pack->primary, m) != 1) { + ksdebug(pack, "Aborted. Could not add dependency on " + "symbol %s from module %s.\n", sym->name, + m->name); + return UNEXPECTED; + } + } + return OK; +} + +static abort_t finalize_patches(struct ksplice_pack *pack) +{ + struct ksplice_patch *p; + struct safety_record *rec; + abort_t ret; + + for (p = pack->patches; p < pack->patches_end; p++) { + struct labelval *lv = find_labelval(pack, p->label); + bool found = false; + if (lv == NULL) { + ksdebug(pack, "Failed to find %s for oldaddr\n", + p->label); + return FAILED_TO_FIND; + } + p->trampoline.oldaddr = lv->val; + + list_for_each_entry(rec, &pack->safety_records, list) { + if (strcmp(rec->label, p->label) == 0 && + follow_trampolines(pack, p->trampoline.oldaddr) + == rec->addr) { + found = true; + break; + } + } + if (!found) { + ksdebug(pack, "No safety record for patch %s\n", + p->label); + return NO_MATCH; + } + if (rec->size < p->trampoline.size) { + ksdebug(pack, "Symbol %s is too short for trampoline\n", + p->label); + return UNEXPECTED; + } + + if (p->trampoline.repladdr == 0) + p->trampoline.repladdr = (unsigned long)ksplice_deleted; + else + rec->first_byte_safe = true; + + ret = prepare_trampoline(pack, &p->trampoline); + if (ret != OK) + return ret; + + if (p->trampoline.oldaddr != rec->addr) { + /* If there's already a trampoline at oldaddr, prepare + a reverse trampoline to install there */ + p->reverse_trampoline.oldaddr = rec->addr; + p->reverse_trampoline.repladdr = p->trampoline.oldaddr; + ret = prepare_trampoline(pack, &p->reverse_trampoline); + if (ret != OK) + return ret; + } else { + p->reverse_trampoline.size = 0; + } + + ret = add_dependency_on_address(pack, p->trampoline.oldaddr); + if (ret != OK) + return ret; + } + return OK; +} + +static abort_t add_dependency_on_address(struct ksplice_pack *pack, + unsigned long addr) +{ + struct module *m = + __module_text_address(follow_trampolines(pack, addr)); + if (m == NULL || m == pack->primary) + return OK; + if (use_module(pack->primary, m) != 1) + return MODULE_BUSY; + return OK; +} + +static abort_t apply_relocs(struct ksplice_pack *pack, + const struct ksplice_reloc *relocs, + const struct ksplice_reloc *relocs_end) +{ + const struct ksplice_reloc *r; + for (r = relocs; r < relocs_end; r++) { + abort_t ret = apply_reloc(pack, r); + if (ret != OK) + return ret; + } + return OK; +} + +static abort_t apply_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r) +{ + abort_t ret; + int canary_ret; + unsigned long sym_addr; + LIST_HEAD(vals); + + canary_ret = contains_canary(pack, r->blank_addr, r->size, r->dst_mask); + if (canary_ret < 0) + return UNEXPECTED; + if (canary_ret == 0) { + ksdebug(pack, "reloc: skipped %s:%lx (altinstr)\n", + r->symbol->label, r->blank_offset); + return OK; + } + + ret = lookup_symbol(pack, r->symbol, &vals); + if (ret != OK) { + release_vals(&vals); + return ret; + } + if (!singular(&vals)) { + release_vals(&vals); + ksdebug(pack, "Failed to find %s for reloc\n", + r->symbol->label); + return FAILED_TO_FIND; + } + sym_addr = list_entry(vals.next, struct candidate_val, list)->val; + release_vals(&vals); + + ret = write_reloc_value(pack, r, r->blank_addr, + r->pcrel ? sym_addr - r->blank_addr : sym_addr); + if (ret != OK) + return ret; + + ksdebug(pack, "reloc: %s:%lx", r->symbol->label, r->blank_offset); + ksdebug(pack, "(S=%lx A=%lx ", sym_addr, r->addend); + switch (r->size) { + case 1: + ksdebug(pack, "aft=%02x)\n", *(uint8_t *)r->blank_addr); + break; + case 2: + ksdebug(pack, "aft=%04x)\n", *(uint16_t *)r->blank_addr); + break; + case 4: + ksdebug(pack, "aft=%08x)\n", *(uint32_t *)r->blank_addr); + break; +#if BITS_PER_LONG >= 64 + case 8: + ksdebug(pack, "aft=%016llx)\n", *(uint64_t *)r->blank_addr); + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + /* Create labelvals so that we can verify our choices in the second + round of run-pre matching that considers data sections. */ + ret = create_labelval(pack, r->symbol->label, sym_addr, VAL); + if (ret != OK) + return ret; + return add_dependency_on_address(pack, sym_addr); +} + +static abort_t read_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long *valp) +{ + unsigned char bytes[sizeof(long)]; + unsigned long val; + + if (probe_kernel_read(bytes, (void *)addr, r->size) == -EFAULT) + return NO_MATCH; + + switch (r->size) { + case 1: + val = *(uint8_t *)bytes; + break; + case 2: + val = *(uint16_t *)bytes; + break; + case 4: + val = *(uint32_t *)bytes; + break; +#if BITS_PER_LONG >= 64 + case 8: + val = *(uint64_t *)bytes; + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + + val &= r->dst_mask; + if (r->signed_addend) + val |= -(val & (r->dst_mask & ~(r->dst_mask >> 1))); + val <<= r->rightshift; + val -= r->addend; + *valp = val; + return OK; +} + +static abort_t write_reloc_value(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long addr, unsigned long sym_addr) +{ + unsigned long val = sym_addr + r->addend; + val >>= r->rightshift; + switch (r->size) { + case 1: + *(uint8_t *)addr = + (*(uint8_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; + case 2: + *(uint16_t *)addr = + (*(uint16_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; + case 4: + *(uint32_t *)addr = + (*(uint32_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; +#if BITS_PER_LONG >= 64 + case 8: + *(uint64_t *)addr = + (*(uint64_t *)addr & ~r->dst_mask) | (val & r->dst_mask); + break; +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return UNEXPECTED; + } + + if (read_reloc_value(pack, r, addr, &val) != OK || val != sym_addr) { + ksdebug(pack, "Aborted. Relocation overflow.\n"); + return UNEXPECTED; + } + + return OK; +} + +static void __attribute__((noreturn)) ksplice_deleted(void) +{ + printk(KERN_CRIT "Called a kernel function deleted by Ksplice!\n"); + BUG(); +} + +static abort_t match_pack_sections(struct ksplice_pack *pack, + bool consider_data_sections) +{ + const struct ksplice_section *sect; + abort_t ret; + char *finished; + int i, remaining = 0; + bool progress; + + finished = kcalloc(pack->helper_sections_end - pack->helper_sections, + sizeof(*finished), GFP_KERNEL); + if (finished == NULL) + return OUT_OF_MEMORY; + for (sect = pack->helper_sections; sect < pack->helper_sections_end; + sect++) { + if ((sect->flags & KSPLICE_SECTION_DATA) == 0) + remaining++; + } + + while (remaining > 0) { + progress = false; + for (sect = pack->helper_sections; + sect < pack->helper_sections_end; sect++) { + i = sect - pack->helper_sections; + if (finished[i]) + continue; + if (!consider_data_sections && + (sect->flags & KSPLICE_SECTION_DATA) != 0) + continue; + ret = find_section(pack, sect); + if (ret == OK) { + finished[i] = 1; + if ((sect->flags & KSPLICE_SECTION_DATA) == 0) + remaining--; + progress = true; + } else if (ret != NO_MATCH) { + kfree(finished); + return ret; + } + } + + if (progress) + continue; + + for (sect = pack->helper_sections; + sect < pack->helper_sections_end; sect++) { + i = sect - pack->helper_sections; + if (finished[i] == 0) + ksdebug(pack, "run-pre: could not match " + "section %s\n", sect->symbol->label); + } + ksdebug(pack, "Aborted. run-pre: could not match some " + "sections.\n"); + kfree(finished); + return NO_MATCH; + } + kfree(finished); + return OK; +} + +static abort_t find_section(struct ksplice_pack *pack, + const struct ksplice_section *sect) +{ + int i; + abort_t ret; + unsigned long run_addr; + LIST_HEAD(vals); + struct candidate_val *v, *n; + + ret = lookup_symbol(pack, sect->symbol, &vals); + if (ret != OK) { + release_vals(&vals); + return ret; + } + + ksdebug(pack, "run-pre: starting sect search for %s\n", + sect->symbol->label); + + list_for_each_entry_safe(v, n, &vals, list) { + run_addr = v->val; + + yield(); + ret = try_addr(pack, sect, run_addr, NULL, RUN_PRE_INITIAL); + if (ret == NO_MATCH) { + list_del(&v->list); + kfree(v); + } else if (ret != OK) { + release_vals(&vals); + return ret; + } + } + + prune_trampoline_vals(pack, &vals); + if (singular(&vals)) { + LIST_HEAD(safety_records); + run_addr = list_entry(vals.next, struct candidate_val, + list)->val; + ret = try_addr(pack, sect, run_addr, &safety_records, + RUN_PRE_FINAL); + release_vals(&vals); + if (ret != OK) { + clear_list(&safety_records, struct safety_record, list); + ksdebug(pack, "run-pre: Final run failed for sect " + "%s:\n", sect->symbol->label); + } else { + list_splice(&safety_records, &pack->safety_records); + } + return ret; + } else if (!list_empty(&vals)) { + struct candidate_val *val; + ksdebug(pack, "run-pre: multiple candidates for sect %s:\n", + sect->symbol->label); + i = 0; + list_for_each_entry(val, &vals, list) { + i++; + ksdebug(pack, "%lx\n", val->val); + if (i > 5) { + ksdebug(pack, "...\n"); + break; + } + } + release_vals(&vals); + return NO_MATCH; + } + release_vals(&vals); + return NO_MATCH; +} + +static abort_t try_addr(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode) +{ + abort_t ret; + const struct module *run_module; + + if ((sect->flags & KSPLICE_SECTION_RODATA) != 0 || + (sect->flags & KSPLICE_SECTION_DATA) != 0) + run_module = __module_data_address(run_addr); + else + run_module = __module_text_address(run_addr); + if (!patches_module(run_module, pack->target)) { + ksdebug(pack, "run-pre: ignoring address %lx in other module " + "%s for sect %s\n", run_addr, run_module == NULL ? + "vmlinux" : run_module->name, sect->symbol->label); + return NO_MATCH; + } + + ret = create_labelval(pack, sect->symbol->label, run_addr, TEMP); + if (ret != OK) + return ret; + + ret = run_pre_cmp(pack, sect, run_addr, safety_records, mode); + if (ret == NO_MATCH && mode != RUN_PRE_FINAL) { + set_temp_labelvals(pack, NOVAL); + ksdebug(pack, "run-pre: %s sect %s does not match (r_a=%lx " + "p_a=%lx s=%lx)\n", + (sect->flags & KSPLICE_SECTION_RODATA) != 0 ? "data" : + "text", sect->symbol->label, run_addr, sect->address, + sect->size); + ksdebug(pack, "run-pre: "); + if (pack->update->debug >= 1) { + ret = run_pre_cmp(pack, sect, run_addr, safety_records, + RUN_PRE_DEBUG); + set_temp_labelvals(pack, NOVAL); + } + ksdebug(pack, "\n"); + return ret; + } else if (ret != OK) { + set_temp_labelvals(pack, NOVAL); + return ret; + } + + if (mode != RUN_PRE_FINAL) { + set_temp_labelvals(pack, NOVAL); + ksdebug(pack, "run-pre: candidate for sect %s=%lx\n", + sect->symbol->label, run_addr); + return OK; + } + + set_temp_labelvals(pack, VAL); + ksdebug(pack, "run-pre: found sect %s=%lx\n", sect->symbol->label, + run_addr); + return OK; +} + +static abort_t run_pre_cmp(struct ksplice_pack *pack, + const struct ksplice_section *sect, + unsigned long run_addr, + struct list_head *safety_records, + enum run_pre_mode mode) +{ + int matched = 0; + abort_t ret; + const struct ksplice_reloc *r; + const unsigned char *pre, *run, *pre_start, *run_start; + unsigned char runval; + + if ((sect->flags & KSPLICE_SECTION_TEXT) != 0) + run_addr = follow_trampolines(pack, run_addr); + + pre_start = (const unsigned char *)sect->address; + run_start = (const unsigned char *)run_addr; + + pre = pre_start; + run = run_start; + while (pre < pre_start + sect->size) { + unsigned long offset = pre - pre_start; + ret = lookup_reloc(pack, (unsigned long)pre, &r); + if (ret == OK) { + ret = handle_reloc(pack, r, (unsigned long)run, mode); + if (ret != OK) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "reloc in sect does not " + "match after %lx/%lx bytes\n", + offset, sect->size); + return ret; + } + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, r->size, pre, r->size); + pre += r->size; + run += r->size; + continue; + } else if (ret != NO_MATCH) { + return ret; + } + + if ((sect->flags & KSPLICE_SECTION_TEXT) != 0) { + ret = handle_paravirt(pack, (unsigned long)pre, + (unsigned long)run, &matched); + if (ret != OK) + return ret; + if (matched != 0) { + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, matched, pre, + matched); + pre += matched; + run += matched; + continue; + } + } + + if (probe_kernel_read(&runval, (void *)run, 1) == -EFAULT) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "sect unmapped after %lx/%lx " + "bytes\n", offset, sect->size); + return NO_MATCH; + } + + if (runval != *pre && + (sect->flags & KSPLICE_SECTION_DATA) == 0) { + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "sect does not match after " + "%lx/%lx bytes\n", offset, sect->size); + if (mode == RUN_PRE_DEBUG) { + print_bytes(pack, run, 1, pre, 1); + ksdebug(pack, "[p_o=%lx] ! ", offset); + print_bytes(pack, run + 1, 2, pre + 1, 2); + } + return NO_MATCH; + } + if (mode == RUN_PRE_DEBUG) + print_bytes(pack, run, 1, pre, 1); + pre++; + run++; + } + return create_safety_record(pack, sect, safety_records, run_addr, + run - run_start); +} + +static void print_bytes(struct ksplice_pack *pack, + const unsigned char *run, int runc, + const unsigned char *pre, int prec) +{ + int o; + int matched = min(runc, prec); + for (o = 0; o < matched; o++) { + if (run[o] == pre[o]) + ksdebug(pack, "%02x ", run[o]); + else + ksdebug(pack, "%02x/%02x ", run[o], pre[o]); + } + for (o = matched; o < runc; o++) + ksdebug(pack, "%02x/ ", run[o]); + for (o = matched; o < prec; o++) + ksdebug(pack, "/%02x ", pre[o]); +} + +static abort_t lookup_reloc(struct ksplice_pack *pack, unsigned long addr, + const struct ksplice_reloc **relocp) +{ + const struct ksplice_reloc *r; + int canary_ret; + for (r = pack->helper_relocs; r < pack->helper_relocs_end; r++) { + if (addr >= r->blank_addr && addr < r->blank_addr + r->size) { + canary_ret = contains_canary(pack, r->blank_addr, + r->size, r->dst_mask); + if (canary_ret < 0) + return UNEXPECTED; + if (canary_ret == 0) { + ksdebug(pack, "reloc: skipped %s:%lx " + "(altinstr)\n", r->symbol->label, + r->blank_offset); + return NO_MATCH; + } + if (addr != r->blank_addr) { + ksdebug(pack, "Invalid nonzero relocation " + "offset\n"); + return UNEXPECTED; + } + *relocp = r; + return OK; + } + } + return NO_MATCH; +} + +static abort_t handle_reloc(struct ksplice_pack *pack, + const struct ksplice_reloc *r, + unsigned long run_addr, enum run_pre_mode mode) +{ + unsigned long val; + abort_t ret; + + ret = read_reloc_value(pack, r, run_addr, &val); + if (ret != OK) + return ret; + if (r->pcrel) + val += run_addr; + + if (mode == RUN_PRE_INITIAL) + ksdebug(pack, "run-pre: reloc at r_a=%lx p_a=%lx to %s+%lx: " + "found %s = %lx\n", run_addr, r->blank_addr, + r->symbol->label, r->addend, r->symbol->label, val); + + if (starts_with(r->symbol->label, ".rodata.str")) + return OK; + + if (contains_canary(pack, run_addr, r->size, r->dst_mask) != 0) { + ksdebug(pack, "Aborted. Unexpected canary in run code at %lx" + "\n", run_addr); + return UNEXPECTED; + } + + ret = create_labelval(pack, r->symbol->label, val, TEMP); + if (ret == NO_MATCH && mode == RUN_PRE_INITIAL) { + struct labelval *lv = find_labelval(pack, r->symbol->label); + ksdebug(pack, "run-pre: reloc at r_a=%lx p_a=%lx: labelval %s " + "= %lx(%d) does not match expected %lx\n", run_addr, + r->blank_addr, r->symbol->label, lv->val, lv->status, + val); + } + return ret; +} + +static abort_t lookup_symbol(struct ksplice_pack *pack, + const struct ksplice_symbol *ksym, + struct list_head *vals) +{ + abort_t ret; + struct labelval *lv; + + lv = find_labelval(pack, ksym->label); + if (lv != NULL) { + release_vals(vals); + ksdebug(pack, "using detected sym %s=%lx\n", ksym->label, + lv->val); + return add_candidate_val(vals, lv->val); + } + + if (starts_with(ksym->label, ".rodata.str")) + return OK; + + if (strcmp(ksym->label, "cleanup_module") == 0 && pack->target != NULL + && pack->target->exit != NULL) { + ret = add_candidate_val(vals, + (unsigned long)pack->target->exit); + if (ret != OK) + return ret; + } + + ret = exported_symbol_lookup(ksym->name, vals); + if (ret != OK) + return ret; + + ret = new_export_lookup(pack->update, ksym->name, vals); + if (ret != OK) + return ret; + + ret = lookup_symbol_kallsyms(pack, ksym->name, vals); + if (ret != OK) + return ret; + + return OK; +} + +static abort_t lookup_symbol_kallsyms(struct ksplice_pack *pack, + const char *name, struct list_head *vals) +{ + struct accumulate_struct acc = { pack, name, vals }; + return (__force abort_t) + kallsyms_on_each_symbol(accumulate_matching_names, &acc); +} + +static int accumulate_matching_names(void *data, const char *sym_name, + struct module *sym_owner, + unsigned long sym_val) +{ + struct accumulate_struct *acc = data; + if (strcmp(sym_name, acc->desired_name) == 0 && + patches_module(sym_owner, acc->pack->target) && + sym_owner != acc->pack->primary) + return (__force int)add_candidate_val(acc->vals, sym_val); + return (__force int)OK; +} + +static abort_t exported_symbol_lookup(const char *name, struct list_head *vals) +{ + const struct kernel_symbol *sym; + sym = find_symbol(name, NULL, NULL, true, false); + if (sym == NULL) + return OK; + return add_candidate_val(vals, sym->value); +} + +static abort_t new_export_lookup(struct update *update, + const char *name, struct list_head *vals) +{ + struct ksplice_pack *pack; + struct ksplice_export *exp; + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) { + if (strcmp(exp->new_name, name) == 0 && + exp->sym != NULL && + contains_canary(pack, + (unsigned long)&exp->sym->value, + sizeof(unsigned long), -1) == 0) + return add_candidate_val(vals, exp->sym->value); + } + } + return OK; +} + +static abort_t apply_patches(struct update *update) +{ + int i; + abort_t ret; + struct ksplice_pack *pack; + const struct ksplice_section *sect; + + list_for_each_entry(pack, &update->packs, list) { + for (sect = pack->primary_sections; + sect < pack->primary_sections_end; sect++) { + ret = create_safety_record(pack, sect, + &pack->safety_records, + sect->address, sect->size); + if (ret != OK) + return ret; + } + } + + for (i = 0; i < 5; i++) { + cleanup_conflicts(update); + ret = (__force abort_t)stop_machine(__apply_patches, update, + NULL); + if (ret != CODE_BUSY) + break; + set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(msecs_to_jiffies(1000)); + } + + if (ret == CODE_BUSY) { + print_conflicts(update); + _ksdebug(update, "Aborted %s. stack check: to-be-replaced " + "code is busy.\n", update->kid); + } else if (ret == ALREADY_REVERSED) { + _ksdebug(update, "Aborted %s. Ksplice update %s is already " + "reversed.\n", update->kid, update->kid); + } + + if (ret != OK) + return ret; + + _ksdebug(update, "Atomic patch insertion for %s complete\n", + update->kid); + return OK; +} + +static abort_t reverse_patches(struct update *update) +{ + int i; + abort_t ret; + struct ksplice_pack *pack; + + clear_debug_buf(update); + ret = init_debug_buf(update); + if (ret != OK) + return ret; + + _ksdebug(update, "Preparing to reverse %s\n", update->kid); + + for (i = 0; i < 5; i++) { + cleanup_conflicts(update); + clear_list(&update->conflicts, struct conflict, list); + ret = (__force abort_t)stop_machine(__reverse_patches, update, + NULL); + if (ret != CODE_BUSY) + break; + set_current_state(TASK_INTERRUPTIBLE); + schedule_timeout(msecs_to_jiffies(1000)); + } + list_for_each_entry(pack, &update->packs, list) + clear_list(&pack->safety_records, struct safety_record, list); + + if (ret == CODE_BUSY) { + print_conflicts(update); + _ksdebug(update, "Aborted %s. stack check: to-be-reversed " + "code is busy.\n", update->kid); + } else if (ret == MODULE_BUSY) { + _ksdebug(update, "Update %s is in use by another module\n", + update->kid); + } + + if (ret != OK) + return ret; + + _ksdebug(update, "Atomic patch removal for %s complete\n", update->kid); + return OK; +} + +static int __apply_patches(void *updateptr) +{ + struct update *update = updateptr; + struct ksplice_pack *pack; + struct ksplice_patch *p; + struct ksplice_export *exp; + abort_t ret; + + if (update->stage == STAGE_APPLIED) + return (__force int)OK; + + if (update->stage != STAGE_PREPARING) + return (__force int)UNEXPECTED; + + ret = check_each_task(update); + if (ret != OK) + return (__force int)ret; + + list_for_each_entry(pack, &update->packs, list) { + if (try_module_get(pack->primary) != 1) { + struct ksplice_pack *pack1; + list_for_each_entry(pack1, &update->packs, list) { + if (pack1 == pack) + break; + module_put(pack1->primary); + } + return (__force int)UNEXPECTED; + } + } + + update->stage = STAGE_APPLIED; + + list_for_each_entry(pack, &update->packs, list) + list_add(&pack->module_list_entry.list, &ksplice_module_list); + + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) + exp->sym->name = exp->new_name; + } + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + insert_trampoline(&p->trampoline); + insert_trampoline(&p->reverse_trampoline); + } + } + return (__force int)OK; +} + +static int __reverse_patches(void *updateptr) +{ + struct update *update = updateptr; + struct ksplice_pack *pack; + const struct ksplice_patch *p; + struct ksplice_export *exp; + abort_t ret; + + if (update->stage != STAGE_APPLIED) + return (__force int)OK; + + list_for_each_entry(pack, &update->packs, list) { + if (module_refcount(pack->primary) != 1) + return (__force int)MODULE_BUSY; + } + + ret = check_each_task(update); + if (ret != OK) + return (__force int)ret; + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + ret = verify_trampoline(pack, &p->trampoline); + if (ret != OK) + return (__force int)ret; + ret = verify_trampoline(pack, &p->reverse_trampoline); + if (ret != OK) + return (__force int)ret; + } + } + + update->stage = STAGE_REVERSED; + + list_for_each_entry(pack, &update->packs, list) + module_put(pack->primary); + + list_for_each_entry(pack, &update->packs, list) + list_del(&pack->module_list_entry.list); + + list_for_each_entry(pack, &update->packs, list) { + for (exp = pack->exports; exp < pack->exports_end; exp++) + exp->sym->name = exp->saved_name; + } + + list_for_each_entry(pack, &update->packs, list) { + for (p = pack->patches; p < pack->patches_end; p++) { + remove_trampoline(&p->trampoline); + remove_trampoline(&p->reverse_trampoline); + } + } + return (__force int)OK; +} + +static abort_t check_each_task(struct update *update) +{ + const struct task_struct *g, *p; + abort_t status = OK, ret; + do_each_thread(g, p) { + /* do_each_thread is a double loop! */ + ret = check_task(update, p, false); + if (ret != OK) { + check_task(update, p, true); + status = ret; + } + if (ret != OK && ret != CODE_BUSY) + return ret; + } while_each_thread(g, p); + return status; +} + +static abort_t check_task(struct update *update, + const struct task_struct *t, bool rerun) +{ + abort_t status, ret; + struct conflict *conf = NULL; + + if (rerun) { + conf = kmalloc(sizeof(*conf), GFP_ATOMIC); + if (conf == NULL) + return OUT_OF_MEMORY; + conf->process_name = kstrdup(t->comm, GFP_ATOMIC); + if (conf->process_name == NULL) { + kfree(conf); + return OUT_OF_MEMORY; + } + conf->pid = t->pid; + INIT_LIST_HEAD(&conf->stack); + list_add(&conf->list, &update->conflicts); + } + + status = check_address(update, conf, KSPLICE_IP(t)); + if (t == current) { + ret = check_stack(update, conf, task_thread_info(t), + (unsigned long *)__builtin_frame_address(0)); + if (status == OK) + status = ret; + } else if (!task_curr(t)) { + ret = check_stack(update, conf, task_thread_info(t), + (unsigned long *)KSPLICE_SP(t)); + if (status == OK) + status = ret; + } else if (!is_stop_machine(t)) { + status = UNEXPECTED_RUNNING_TASK; + } + return status; +} + +static abort_t check_stack(struct update *update, struct conflict *conf, + const struct thread_info *tinfo, + const unsigned long *stack) +{ + abort_t status = OK, ret; + unsigned long addr; + + while (valid_stack_ptr(tinfo, stack)) { + addr = *stack++; + ret = check_address(update, conf, addr); + if (ret != OK) + status = ret; + } + return status; +} + +static abort_t check_address(struct update *update, + struct conflict *conf, unsigned long addr) +{ + abort_t status = OK, ret; + const struct safety_record *rec; + struct ksplice_pack *pack; + struct conflict_addr *ca = NULL; + + if (conf != NULL) { + ca = kmalloc(sizeof(*ca), GFP_ATOMIC); + if (ca == NULL) + return OUT_OF_MEMORY; + ca->addr = addr; + ca->has_conflict = false; + ca->label = NULL; + list_add(&ca->list, &conf->stack); + } + + list_for_each_entry(pack, &update->packs, list) { + list_for_each_entry(rec, &pack->safety_records, list) { + ret = check_record(ca, rec, addr); + if (ret != OK) + status = ret; + } + } + return status; +} + +static abort_t check_record(struct conflict_addr *ca, + const struct safety_record *rec, unsigned long addr) +{ + if ((addr > rec->addr && addr < rec->addr + rec->size) || + (addr == rec->addr && !rec->first_byte_safe)) { + if (ca != NULL) { + ca->label = rec->label; + ca->has_conflict = true; + } + return CODE_BUSY; + } + return OK; +} + +static bool is_stop_machine(const struct task_struct *t) +{ + const char *num; + if (!starts_with(t->comm, "kstop")) + return false; + num = t->comm + strlen("kstop"); + return num[strspn(num, "0123456789")] == '\0'; +} + +static void cleanup_conflicts(struct update *update) +{ + struct conflict *conf; + list_for_each_entry(conf, &update->conflicts, list) { + clear_list(&conf->stack, struct conflict_addr, list); + kfree(conf->process_name); + } + clear_list(&update->conflicts, struct conflict, list); +} + +static void print_conflicts(struct update *update) +{ + const struct conflict *conf; + const struct conflict_addr *ca; + list_for_each_entry(conf, &update->conflicts, list) { + _ksdebug(update, "stack check: pid %d (%s):", conf->pid, + conf->process_name); + list_for_each_entry(ca, &conf->stack, list) { + _ksdebug(update, " %lx", ca->addr); + if (ca->has_conflict) + _ksdebug(update, " [<-CONFLICT]"); + } + _ksdebug(update, "\n"); + } +} + +static void insert_trampoline(struct ksplice_trampoline *t) +{ + mm_segment_t old_fs = get_fs(); + set_fs(KERNEL_DS); + memcpy((void *)t->saved, (void *)t->oldaddr, t->size); + memcpy((void *)t->oldaddr, (void *)t->trampoline, t->size); + flush_icache_range(t->oldaddr, t->oldaddr + t->size); + set_fs(old_fs); +} + +static abort_t verify_trampoline(struct ksplice_pack *pack, + const struct ksplice_trampoline *t) +{ + if (memcmp((void *)t->oldaddr, (void *)t->trampoline, t->size) != 0) { + ksdebug(pack, "Aborted. Trampoline at %lx has been " + "overwritten.\n", t->oldaddr); + return CODE_BUSY; + } + return OK; +} + +static void remove_trampoline(const struct ksplice_trampoline *t) +{ + mm_segment_t old_fs = get_fs(); + set_fs(KERNEL_DS); + memcpy((void *)t->oldaddr, (void *)t->saved, t->size); + flush_icache_range(t->oldaddr, t->oldaddr + t->size); + set_fs(old_fs); +} + +static struct labelval *find_labelval(struct ksplice_pack *pack, + const char *label) +{ + struct labelval *lv; + list_for_each_entry(lv, &pack->labelvals, list) { + if (strcmp(lv->label, label) == 0) + return lv; + } + return NULL; +} + +static abort_t create_labelval(struct ksplice_pack *pack, const char *label, + unsigned long val, int status) +{ + struct labelval *lv = find_labelval(pack, label); + if (lv != NULL) + return lv->val == val ? OK : NO_MATCH; + + lv = kmalloc(sizeof(*lv), GFP_KERNEL); + if (lv == NULL) + return OUT_OF_MEMORY; + lv->label = label; + lv->val = val; + lv->status = status; + list_add(&lv->list, &pack->labelvals); + return OK; +} + +static abort_t create_safety_record(struct ksplice_pack *pack, + const struct ksplice_section *sect, + struct list_head *record_list, + unsigned long run_addr, + unsigned long run_size) +{ + struct safety_record *rec; + struct ksplice_patch *p; + + if (record_list == NULL) + return OK; + + for (p = pack->patches; p < pack->patches_end; p++) { + if (strcmp(sect->symbol->label, p->label) == 0) + break; + } + if (p >= pack->patches_end) + return OK; + + if ((sect->flags & KSPLICE_SECTION_TEXT) == 0 && + p->trampoline.repladdr != 0) { + ksdebug(pack, "Error: ksplice_patch %s is matched to a " + "non-deleted non-text section!\n", sect->symbol->label); + return UNEXPECTED; + } + + rec = kmalloc(sizeof(*rec), GFP_KERNEL); + if (rec == NULL) + return OUT_OF_MEMORY; + rec->addr = run_addr; + rec->size = run_size; + rec->label = sect->symbol->label; + rec->first_byte_safe = false; + + list_add(&rec->list, record_list); + return OK; +} + +static abort_t add_candidate_val(struct list_head *vals, unsigned long val) +{ + struct candidate_val *tmp, *new; + + list_for_each_entry(tmp, vals, list) { + if (tmp->val == val) + return OK; + } + new = kmalloc(sizeof(*new), GFP_KERNEL); + if (new == NULL) + return OUT_OF_MEMORY; + new->val = val; + list_add(&new->list, vals); + return OK; +} + +/* If there are only two candidates and their addresses are related by + a trampoline, then we have successfully found a function patched by a + previous update. We remove the endpoint of the trampoline from the vals + list, so that this update uses the patched function's original address. */ +static void prune_trampoline_vals(struct ksplice_pack *pack, + struct list_head *vals) +{ + struct candidate_val *val1, *val2; + + if (list_empty(vals) || singular(vals)) + return; + if (vals->next->next->next != vals) + return; + + val1 = list_entry(vals->next, struct candidate_val, list); + val2 = list_entry(vals->next->next, struct candidate_val, list); + + if (val1->val == follow_trampolines(pack, val2->val)) { + list_del(&val1->list); + kfree(val1); + return; + } + if (val2->val == follow_trampolines(pack, val1->val)) { + list_del(&val2->list); + kfree(val2); + return; + } +} + +static void release_vals(struct list_head *vals) +{ + clear_list(vals, struct candidate_val, list); +} + +static void set_temp_labelvals(struct ksplice_pack *pack, int status) +{ + struct labelval *lv, *n; + list_for_each_entry_safe(lv, n, &pack->labelvals, list) { + if (lv->status == TEMP) { + if (status == NOVAL) { + list_del(&lv->list); + kfree(lv); + } else { + lv->status = status; + } + } + } +} + +static int contains_canary(struct ksplice_pack *pack, unsigned long blank_addr, + int size, long dst_mask) +{ + switch (size) { + case 1: + return (*(uint8_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); + case 2: + return (*(uint16_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); + case 4: + return (*(uint32_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); +#if BITS_PER_LONG >= 64 + case 8: + return (*(uint64_t *)blank_addr & dst_mask) == + (KSPLICE_CANARY & dst_mask); +#endif /* BITS_PER_LONG */ + default: + ksdebug(pack, "Aborted. Invalid relocation size.\n"); + return -1; + } +} + +static unsigned long follow_trampolines(struct ksplice_pack *pack, + unsigned long addr) +{ + unsigned long new_addr; + struct module *m; + + if (trampoline_target(pack, addr, &new_addr) != OK) + return addr; + + /* Confirm that it is a jump into a ksplice module */ + m = __module_text_address(new_addr); + if (m != NULL && m != pack->target && starts_with(m->name, "ksplice")) { + ksdebug(pack, "Following trampoline %lx %lx(%s)\n", addr, + new_addr, m->name); + return new_addr; + } + return addr; +} + +/* Does module a patch module b? */ +static bool patches_module(const struct module *a, const struct module *b) +{ + struct ksplice_module_list_entry *entry; + if (a == b) + return true; + list_for_each_entry(entry, &ksplice_module_list, list) { + if (entry->target == b && entry->primary == a) + return true; + } + return false; +} + +static bool starts_with(const char *str, const char *prefix) +{ + return strncmp(str, prefix, strlen(prefix)) == 0; +} + +static bool singular(struct list_head *list) +{ + return !list_empty(list) && list->next->next == list; +} + +#ifdef CONFIG_DEBUG_FS + +static abort_t init_debug_buf(struct update *update) +{ + update->debug_blob.size = 0; + update->debug_blob.data = NULL; + update->debugfs_dentry = + debugfs_create_blob(update->name, S_IFREG | S_IRUSR, NULL, + &update->debug_blob); + if (update->debugfs_dentry == NULL) + return OUT_OF_MEMORY; + return OK; +} + +static void clear_debug_buf(struct update *update) +{ + if (update->debugfs_dentry == NULL) + return; + debugfs_remove(update->debugfs_dentry); + update->debugfs_dentry = NULL; + update->debug_blob.size = 0; + vfree(update->debug_blob.data); + update->debug_blob.data = NULL; +} + +static int _ksdebug(struct update *update, const char *fmt, ...) +{ + va_list args; + unsigned long size, old_size, new_size; + + if (update->debug == 0) + return 0; + + /* size includes the trailing '\0' */ + va_start(args, fmt); + size = 1 + vsnprintf(update->debug_blob.data, 0, fmt, args); + va_end(args); + old_size = update->debug_blob.size == 0 ? 0 : + max(PAGE_SIZE, roundup_pow_of_two(update->debug_blob.size)); + new_size = update->debug_blob.size + size == 0 ? 0 : + max(PAGE_SIZE, roundup_pow_of_two(update->debug_blob.size + size)); + if (new_size > old_size) { + char *buf = vmalloc(new_size); + if (buf == NULL) + return -ENOMEM; + memcpy(buf, update->debug_blob.data, update->debug_blob.size); + vfree(update->debug_blob.data); + update->debug_blob.data = buf; + } + va_start(args, fmt); + update->debug_blob.size += vsnprintf(update->debug_blob.data + + update->debug_blob.size, + size, fmt, args); + va_end(args); + return 0; +} +#else /* CONFIG_DEBUG_FS */ +static abort_t init_debug_buf(struct update *update) +{ + return OK; +} + +static void clear_debug_buf(struct update *update) +{ + return; +} + +static int _ksdebug(struct update *update, const char *fmt, ...) +{ + va_list args; + + if (update->debug == 0) + return 0; + + if (!update->debug_continue_line) + printk(KERN_DEBUG "ksplice: "); + + va_start(args, fmt); + vprintk(fmt, args); + va_end(args); + + update->debug_continue_line = + fmt[0] == '\0' || fmt[strlen(fmt) - 1] != '\n'; + return 0; +} +#endif /* CONFIG_DEBUG_FS */ + +struct ksplice_attribute { + struct attribute attr; + ssize_t (*show)(struct update *update, char *buf); + ssize_t (*store)(struct update *update, const char *buf, size_t len); +}; + +static ssize_t ksplice_attr_show(struct kobject *kobj, struct attribute *attr, + char *buf) +{ + struct ksplice_attribute *attribute = + container_of(attr, struct ksplice_attribute, attr); + struct update *update = container_of(kobj, struct update, kobj); + if (attribute->show == NULL) + return -EIO; + return attribute->show(update, buf); +} + +static ssize_t ksplice_attr_store(struct kobject *kobj, struct attribute *attr, + const char *buf, size_t len) +{ + struct ksplice_attribute *attribute = + container_of(attr, struct ksplice_attribute, attr); + struct update *update = container_of(kobj, struct update, kobj); + if (attribute->store == NULL) + return -EIO; + return attribute->store(update, buf, len); +} + +static struct sysfs_ops ksplice_sysfs_ops = { + .show = ksplice_attr_show, + .store = ksplice_attr_store, +}; + +static void ksplice_release(struct kobject *kobj) +{ + struct update *update; + update = container_of(kobj, struct update, kobj); + cleanup_ksplice_update(update); +} + +static ssize_t stage_show(struct update *update, char *buf) +{ + switch (update->stage) { + case STAGE_PREPARING: + return snprintf(buf, PAGE_SIZE, "preparing\n"); + case STAGE_APPLIED: + return snprintf(buf, PAGE_SIZE, "applied\n"); + case STAGE_REVERSED: + return snprintf(buf, PAGE_SIZE, "reversed\n"); + } + return 0; +} + +static ssize_t abort_cause_show(struct update *update, char *buf) +{ + switch (update->abort_cause) { + case OK: + return snprintf(buf, PAGE_SIZE, "ok\n"); + case NO_MATCH: + return snprintf(buf, PAGE_SIZE, "no_match\n"); + case CODE_BUSY: + return snprintf(buf, PAGE_SIZE, "code_busy\n"); + case MODULE_BUSY: + return snprintf(buf, PAGE_SIZE, "module_busy\n"); + case OUT_OF_MEMORY: + return snprintf(buf, PAGE_SIZE, "out_of_memory\n"); + case FAILED_TO_FIND: + return snprintf(buf, PAGE_SIZE, "failed_to_find\n"); + case ALREADY_REVERSED: + return snprintf(buf, PAGE_SIZE, "already_reversed\n"); + case MISSING_EXPORT: + return snprintf(buf, PAGE_SIZE, "missing_export\n"); + case UNEXPECTED_RUNNING_TASK: + return snprintf(buf, PAGE_SIZE, "unexpected_running_task\n"); + case UNEXPECTED: + return snprintf(buf, PAGE_SIZE, "unexpected\n"); + } + return 0; +} + +static ssize_t conflict_show(struct update *update, char *buf) +{ + const struct conflict *conf; + const struct conflict_addr *ca; + int used = 0; + list_for_each_entry(conf, &update->conflicts, list) { + used += snprintf(buf + used, PAGE_SIZE - used, "%s %d", + conf->process_name, conf->pid); + list_for_each_entry(ca, &conf->stack, list) { + if (!ca->has_conflict) + continue; + used += snprintf(buf + used, PAGE_SIZE - used, " %s", + ca->label); + } + used += snprintf(buf + used, PAGE_SIZE - used, "\n"); + } + return used; +} + +static ssize_t stage_store(struct update *update, const char *buf, size_t len) +{ + if ((strncmp(buf, "applied", len) == 0 || + strncmp(buf, "applied\n", len) == 0) && + update->stage == STAGE_PREPARING) + update->abort_cause = apply_update(update); + else if ((strncmp(buf, "reversed", len) == 0 || + strncmp(buf, "reversed\n", len) == 0) && + update->stage == STAGE_APPLIED) + update->abort_cause = reverse_patches(update); + if (update->abort_cause == OK) + printk(KERN_INFO "ksplice: Update %s %s successfully\n", + update->kid, + update->stage == STAGE_APPLIED ? "applied" : "reversed"); + return len; +} + +static ssize_t debug_show(struct update *update, char *buf) +{ + return snprintf(buf, PAGE_SIZE, "%d\n", update->debug); +} + +static ssize_t debug_store(struct update *update, const char *buf, size_t len) +{ + unsigned long l; + int ret = strict_strtoul(buf, 10, &l); + if (ret != 0) + return ret; + update->debug = l; + return len; +} + +static struct ksplice_attribute stage_attribute = + __ATTR(stage, 0600, stage_show, stage_store); +static struct ksplice_attribute abort_cause_attribute = + __ATTR(abort_cause, 0400, abort_cause_show, NULL); +static struct ksplice_attribute debug_attribute = + __ATTR(debug, 0600, debug_show, debug_store); +static struct ksplice_attribute conflict_attribute = + __ATTR(conflicts, 0400, conflict_show, NULL); + +static struct attribute *ksplice_attrs[] = { + &stage_attribute.attr, + &abort_cause_attribute.attr, + &debug_attribute.attr, + &conflict_attribute.attr, + NULL +}; + +static struct kobj_type ksplice_ktype = { + .sysfs_ops = &ksplice_sysfs_ops, + .release = ksplice_release, + .default_attrs = ksplice_attrs, +}; + +static int init_ksplice(void) +{ + ksplice_kobj = kobject_create_and_add("ksplice", kernel_kobj); + if (ksplice_kobj == NULL) + return -ENOMEM; + return 0; +} + +static void cleanup_ksplice(void) +{ + kobject_put(ksplice_kobj); +} + +module_init(init_ksplice); +module_exit(cleanup_ksplice); + +MODULE_AUTHOR("Jeffrey Brian Arnold <jbarnold@mit.edu>"); +MODULE_DESCRIPTION("Ksplice rebootless update system"); +MODULE_LICENSE("GPL v2"); -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 6/9] Ksplice: Documentation 2008-09-13 4:49 ` [RFC PATCH 5/9] Ksplice core (architecture independent) Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 0 siblings, 0 replies; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko This document assumes familiarity with the Ksplice design and describes notable implementation details and the interface between the Ksplice kernel component and the Ksplice user space component. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> --- Documentation/ksplice.txt | 247 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 247 insertions(+), 0 deletions(-) create mode 100644 Documentation/ksplice.txt diff --git a/Documentation/ksplice.txt b/Documentation/ksplice.txt new file mode 100644 index 0000000..a8fb0ba --- /dev/null +++ b/Documentation/ksplice.txt @@ -0,0 +1,247 @@ +Ksplice +------- + +CONTENTS: + +1. Concepts: Updates, packs, helper modules, primary modules +2. What changes can Ksplice handle? +3. Dependency model +4. Locking model +5. altinstructions, smplocks, and parainstructions +6. sysfs interface +7. debugfs interface + +0. Design Description +--------------------- + +For a description of the Ksplice design, please see the Ksplice technical +overview document: <http://web.mit.edu/ksplice/doc/ksplice.pdf>. For usage +examples and the Ksplice man pages, please see <http://web.mit.edu/ksplice/>. + +The document below assumes familiarity with the Ksplice design and describes +notable implementation details and the interface between the Ksplice kernel +component and the Ksplice user space component. + +1. Concepts: Updates, packs, helper modules, primary modules +------------------------------------------------------------ + +A Ksplice update (struct update) contains one or more Ksplice packs, one for +each target kernel module that should be changed by the update. Ksplice packs +are grouped together into a Ksplice update in order to allow multiple +compilation units to be changed atomically. + +The contents of a Ksplice pack are documented via kernel-doc in +include/linux/ksplice.h. To construct a new Ksplice update to be performed +atomically, one needs to: + 1. Populate the fields of one or more ksplice_pack structures. + 2. Call the Ksplice function init_ksplice_pack() on each pack to register + the packs with the Ksplice kernel component. When init_ksplice_pack() + is called on a pack, that pack will be associated with the other packs + that share the same Ksplice identifier (KID) field. + 3. After all of the packs intended for a particular Ksplice update have + been loaded, that update can be applied via the sysfs interface + (described in Section 7 below). + +In order to save memory, each Ksplice pack has a "helper" module and a "primary" +module associated with it. + +The pack's "helper" module contains materials needed only for preparing for the +update. Specifically, the helper module contains a copy of the pre-patch +version of each of the compilation units changed by the Ksplice pack. The +helper module can be unloaded after the update has been applied. + +The pack's "primary" module contains the new sections to be inserted by the +update; it needs to remain loaded for as long as the update is applied. + +Here's an example: + +Let's say that the Ksplice user space component wants to update the core kernel +and the isdn module. The user space component will select a KID for this update +(let's say 123abc) and generate four modules: + +ksplice_123abc_vmlinux (the "primary" module for the vmlinux pack) +ksplice_123abc_vmlinux_helper (the "helper" module for the vmlinux pack) +ksplice_123abc_isdn (the "primary" module for the vmlinux pack) +ksplice_123abc_isdn_helper (the "helper" module for the vmlinux pack) + +Once both of the vmlinux modules have been loaded, one of the modules calls +init_ksplice_pack on a pack corresponding to the desired vmlinux changes. + +Similarly, once both of the isdn modules have been loaded, one of the modules +calls init_ksplice_pack on a pack corresponding to the desired isdn changes. + +Once all modules are loaded (in this example, four modules), the update can be +applied atomically using the Ksplice sysfs interface. Once the update has been +applied, the helper modules can be unloaded safely to save memory. + +2. What changes can Ksplice handle? +----------------------------------- + +The Ksplice user space component takes a source code patch and uses it to +construct appropriate Ksplice packs for an update. Ksplice can handle source +code patches that add new functions, modify the text or arguments of existing +functions, delete functions, change functions from local to global (or vice +versa), add exported symbols, rename exported symbols, and delete exported +symbols. Ksplice can handle patches that modify either C code or assembly code. + +As described in the Ksplice technical overview document, Ksplice cannot handle +semantic changes to kernel data structures. Some other limitations also apply: + +Ksplice does not support changes to __init functions that been unloaded from +kernel memory. Ksplice also does not support changes to functions in .exit.text +sections since Ksplice currently requires that all Ksplice updates affecting a +module be reversed before that module can be unloaded. + +The Ksplice user space implementation does not currently support moving +functions between compilation units, changes to weak symbols, and changes to +global read-only data structures (changes to read-only data structures that are +local to a compilation unit are fine). + +Exported symbols: + +Ksplice can handle arbitrary changes to exported symbols in the source code +patch. + +Ksplice deletes exported symbols by looking up the relevant struct kernel_symbol +in the kernel's exported symbol table and replacing the name field with a +pointer to a string that begins with DISABLED. + +Ksplice adds new exported symbols through the same mechanism; the relevant +primary module will have a ksymtab entry containing a symbol with a name +beginning with DISABLED, and Ksplice will replace that with the name of the +symbol to be exported when the update is atomically applied. + +Because the struct kernel_symbol for a newly exported symbol is contained in the +Ksplice primary module, if a module using one of the newly exported symbols is +loaded, that module will correctly depend on the Ksplice primary module that +exported the symbol. + +3. Dependency model +------------------- + +Because Ksplice resolves symbols used in the post code using Ksplice +relocations, Ksplice must enforce additional dependencies. Ksplice uses the +use_module function to directly add dependencies on all the modules that the +post code references. + +4. Locking model +---------------- + +From a locking perspective, Ksplice treats applying or removing a Ksplice update +as analogous to loading or unloading a new version of the kernel modules patched +by the update. Ksplice uses module_mutex to protect against a variety of race +conditions related to modules being loaded or unloaded while Ksplice is applying +or reversing an update; this approach also protects against race conditions +involving multiple Ksplice updates being loaded or unloaded simultaneously as +well. + +5. altinstructions, smplocks, and parainstructions +-------------------------------------------------- + +There are currently several mechanisms through which the Linux kernel will +modify executable code at runtime. + +These mechanisms sometimes overwrite the storage unit of a relocation, which +would cause problems if not handled properly by Ksplice. + +Ksplice solves this problem by writing "canary" bytes (e.g., 0x77777777) in the +storage unit of the relocation in user space. Ksplice then checks whether the +canary has been overwritten before using a Ksplice relocation to detect symbol +values or to write a value to the storage unit of a Ksplice relocation. + +6. sysfs interface +------------------ + +Ksplice exports four sysfs files per Ksplice update in order to communicate with +user space. For each update, these four files are located in a directory of the +form /sys/kernel/ksplice/$kid, with $kid replaced by the KID of the Ksplice +update. + +A. /sys/kernel/ksplice/$kid/stage (mode 0600) + +This file contains one of three strings: +preparing: Indicates that this update has not yet been applied +applied: Indicates that this update has been applied and has not been reversed +reversed: Indicates that this update has been reversed + +When the stage is "preparing", the superuser can write "applied" to the stage +file in order to instruct Ksplice to apply the update. When the stage is +"applied", the superuser can write "reversed" to the stage file in order to +instruct Ksplice to reverse the update. After an update has been reversed, its +stage cannot change again (although the update can be removed from the kernel +entirely, in which case its $kid directory will be removed). + +B. /sys/kernel/ksplice/$kid/debug (mode 0600) + +The file contains a single number: 1 if debugging is enabled for this Ksplice +update and 0 otherwise. + +The superuser can write a new value to this file to enable or disable debugging. + +C. /sys/kernel/ksplice/$kid/abort_cause (mode 0400) + +This file contains a value indicating either 1) that Ksplice successfully +completed the most recently requested stage transition or 2) why Ksplice aborted +the most recently requested stage transition. + +Each abort_code string is described below, along with the stage transitions that +might potentially trigger each possible abort code. The stage transitions are +abbreviated as follows: preparing->applied (P->A), applied->reversed (A->R). + +ok (P->A, A->R): The most recent stage transition succeeded. + +no_match (P->A): Ksplice aborted the update because Ksplice was unable to match +the helper module's object code against the running kernel's object code. + +failed_to_find (P->A): Ksplice aborted the update because Ksplice was unable to +resolve some of the symbols used in the update. + +missing_export (P->A): Ksplice aborted the update because the symbols exported +by the kernel did not match Ksplice's expectations based on the ksplice_export +structures provided to Ksplice. + +already_reversed (P->A): Ksplice aborted the update because once an update has +been reversed, it cannot be applied again (without first being cleaned up and +reinitialized). + +module_busy (A->R): Ksplice aborted the undo operation because the target +Ksplice update is in use by another kernel module; specifically, either the +target Ksplice update exports a symbol that is in use by another module or +another Ksplice update depends on this Ksplice update. + +out_of_memory (P->A, A->R): Ksplice aborted the operation because a call to +kmalloc or vmalloc failed. + +code_busy (P->A, A->R): Ksplice aborted the operation because Ksplice was +unable to find a moment when one or more of the to-be-patched functions was not +a thread's kernel stack. + +unexpected_running_task (P->A, A->R): Ksplice aborted the operation because +Ksplice observed a running task during the kernel stack check, at a time when +Ksplice expected all tasks to be stopped by stop_machine. + +unexpected (P->A, A->R): Ksplice aborted the operation because it encountered +an unspecified internal error. This condition can only be caused by an invalid +input to Ksplice or a bug in Ksplice. + +D. /sys/kernel/ksplice/$kid/conflicts (mode 0400) + +This file is empty until Ksplice aborts an operation because of a code_busy +condition (see "abort_code" above). This conflicts file then contains +information about the process(es) that caused the stack check failure. + +Specifically, each line of this file consists of three space-separated values, +describing a single conflict: + +$program_name $program_pid $conflict_label + +$program_name is the name of the program with the conflict. +$program_pid is the pid of the program with the conflict. +$conflict_label is the Ksplice label of the function with the conflict. + +7. debugfs interface +-------------------- + +Ksplice exports a single file to debugfs for each Ksplice update. The file has +a name of the form ksplice_KID, where KID is the unique identifier of the +Ksplice update. It contains debugging information in a human-readable format. -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 6/9] Ksplice documentation [not found] ` <8d86b7714a1012014ce3533aa24dc88704fe14a6.1221271644.git.jbarnold@mit.edu> 2008-09-13 4:49 ` [RFC PATCH 5/9] Ksplice core (architecture independent) Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 7/9] Ksplice: x86-32 and x86-64 support Jeffrey Brian Arnold 1 sibling, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko This document assumes familiarity with the Ksplice design and describes notable implementation details and the interface between the Ksplice kernel component and the Ksplice user space component. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> --- Documentation/ksplice.txt | 251 +++++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 251 insertions(+), 0 deletions(-) create mode 100644 Documentation/ksplice.txt diff --git a/Documentation/ksplice.txt b/Documentation/ksplice.txt new file mode 100644 index 0000000..5ec9173 --- /dev/null +++ b/Documentation/ksplice.txt @@ -0,0 +1,251 @@ +Ksplice +------- + +CONTENTS: + +1. Concepts: updates, packs, helper modules, primary modules +2. What changes can Ksplice handle? +3. Dependency model +4. Locking model +5. altinstructions, smplocks, and parainstructions +6. sysfs interface +7. debugfs interface + +0. Design Description +--------------------- + +For a description of the Ksplice design, please see the Ksplice technical +overview document: <http://web.mit.edu/ksplice/doc/ksplice.pdf>. For usage +examples and the Ksplice man pages, please see <http://web.mit.edu/ksplice/>. + +The document below assumes familiarity with the Ksplice design and describes +notable implementation details and the interface between the Ksplice kernel +component and the Ksplice user space component. + +1. Updates, packs, helper modules, primary modules +-------------------------------------------------- + +A Ksplice update (struct update) contains one or more Ksplice packs, one for +each "target" kernel module that should be changed by the update. Ksplice packs +are grouped together into a Ksplice update in order to allow multiple +compilation units to be changed atomically. + +The contents of a Ksplice pack are documented via kernel-doc in +include/linux/ksplice.h. To construct a new Ksplice update to be performed +atomically, one needs to: + 1. Populate the fields of one or more ksplice_pack structures. + 2. Call the Ksplice function init_ksplice_pack() on each pack to register + the packs with the Ksplice kernel component. When init_ksplice_pack() + is called on a pack, that pack will be associated with the other packs + that share the same Ksplice identifier (KID) field. + 3. After all of the packs intended for a particular Ksplice update have + been loaded, that update can be applied via the sysfs interface + (described in Section 7 below). + +In order to save memory, each Ksplice pack has a "helper" module and a "primary" +module associated with it. + +The pack's "helper" module contains materials needed only for preparing for the +update. Specifically, the helper module contains a copy of the pre-patch +version of each of the compilation units changed by the Ksplice pack. The +helper module can be unloaded after the update has been applied. + +The pack's "primary" module contains the new sections to be inserted by the +update; it needs to remain loaded for as long as the update is applied. + +Here's an example: + +Let's say that the Ksplice user space component wants to update the core kernel +and the isdn module. The user space component will select a KID for this update +(let's say 123abc) and generate four modules: + +ksplice_123abc_vmlinux (the "primary" module for the vmlinux pack) +ksplice_123abc_vmlinux_helper (the "helper" module for the vmlinux pack) +ksplice_123abc_isdn (the "primary" module for the vmlinux pack) +ksplice_123abc_isdn_helper (the "helper" module for the vmlinux pack) + +Once both of the vmlinux modules have been loaded, one of the modules calls +init_ksplice_pack on a pack corresponding to the desired vmlinux changes. + +Similarly, once both of the isdn modules have been loaded, one of the modules +calls init_ksplice_pack on a pack corresponding to the desired isdn changes. + +Once all modules are loaded (in this example, four modules), the update can be +applied atomically using the Ksplice sysfs interface. Once the update has been +applied, the helper modules can be unloaded safely to save memory. + +2. What changes can Ksplice handle? +----------------------------------- + +The Ksplice user space component takes a source code patch and uses it to +construct appropriate Ksplice packs for an update. Ksplice can handle source +code patches that add new functions, modify the text or arguments of existing +functions, delete functions, move functions between compilation units, change +functions from local to global (or vice versa), add exported symbols, rename +exported symbols, delete exported symbols. Ksplice can handle patches that +modify either C code or assembly code. + +As described in the Ksplice technical overview document, Ksplice cannot handle +semantic changes to kernel data structures. Some other limitations also apply: + +Ksplice does not support changes to __init functions that been unloaded from +kernel memory. Ksplice also does not support changes to functions in .exit.text +sections since Ksplice currently requires that all Ksplice updates affecting a +module be reversed before that module can be unloaded. + +Exported symbols: + +Ksplice can handle arbitrary changes to exported symbols in the source code +patch. + +Ksplice deletes exported symbols by looking up the relevant struct kernel_symbol +in the kernel's exported symbol table and replacing the name field with a +pointer to a string that begins with DISABLED. + +Ksplice adds new exported symbols through the same mechanism; the relevant +primary module will have a ksymtab entry containing a symbol with a name +beginning with DISABLED, and Ksplice will replace that with the name of the +symbol to be exported when the update is atomically applied. + +Because the struct kernel_symbol for a newly exported symbol is contained in the +Ksplice primary module, if a module using one of the newly exported symbols is +loaded, that module will correctly depend on the Ksplice primary module that +exported the symbol. + +Read-only data: + +The current Ksplice user space implementation does not support changes to global +read-only data structures (changes to read-only data structures that are local +to a compilation unit are fine). The user space implementation currently +handles changes to read-only data structures by patching all functions that +reference those data structures. + +3. Dependency model +------------------- + +Because Ksplice resolves symbols used in the post code using Ksplice +relocations, Ksplice must enforce additional dependencies. Ksplice uses the +use_module function to directly add dependencies on all the modules that the +post code references. + +4. Locking model +---------------- + +From a locking perspective, Ksplice treats applying or removing a Ksplice update +as analogous to loading or unloading a new version of the kernel modules patched +by the update. Ksplice uses module_mutex to protect against a variety of race +conditions related to modules being loaded or unloaded while Ksplice is applying +or reversing an update; this approach also protects against race conditions +involving multiple Ksplice updates being loaded or unloaded simultaneously as +well. + +5. altinstructions, smplocks, and parainstructions +-------------------------------------------------- + +There are currently several mechanisms through which the Linux kernel will +modify executable code at runtime. + +These mechanisms sometimes overwrite the storage unit of a relocation, which +would cause problems if not handled properly by Ksplice. + +Ksplice solves this problem by writing "canary" bytes (e.g., 0x77777777) in the +storage unit of the relocation in user space. Ksplice then checks whether the +canary has been overwritten before using a Ksplice relocation to detect symbol +values or to write a value to the storage unit of a Ksplice relocation. + +6. sysfs interface +------------------ + +Ksplice exports four sysfs files per Ksplice update in order to communicate with +user space. For each update, these four files are located in a directory of the +form /sys/kernel/ksplice/$kid, with $kid replaced by the KID of the Ksplice +update. + +A. /sys/kernel/ksplice/$kid/stage (mode 0600) + +This file contains one of three strings: +preparing: Indicates that this update has not yet been applied +applied: Indicates that this update has been applied and has not been reversed +reversed: Indicates that this update has been reversed + +When the stage is "preparing", the superuser can write "applied" to the stage +file in order to instruct Ksplice to apply the update. When the stage is +"applied", the superuser can write "reversed" to the stage file in order to +instruct Ksplice to reverse the update. After an update has been reversed, its +stage cannot change again (although the update can be removed from the kernel +entirely, in which case its $kid directory will be removed). + +B. /sys/kernel/ksplice/$kid/debug (mode 0600) + +The file contains a single number: 1 if debugging is enabled for this Ksplice +update and 0 otherwise. + +The superuser can write a new value to this file to enable or disable debugging. + +C. /sys/kernel/ksplice/$kid/abort_cause (mode 0400) + +This file contains a value indicating either 1) that Ksplice successfully +completed the most recently requested stage transition or 2) why Ksplice aborted +the most recently requested stage transition. + +Each abort_code string is described below, along with the stage transitions that +might potentially trigger each possible abort code. The stage transitions are +abbreviated as follows: preparing->applied (P->A), applied->reversed (A->R). + +ok (P->A, A->R): The most recent stage transition succeeded. + +no_match (P->A): Ksplice aborted the update because Ksplice was unable to match +the helper module's object code against the running kernel's object code. + +failed_to_find (P->A): Ksplice aborted the update because Ksplice was unable to +resolve some of the symbols used in the update. + +missing_export (P->A): Ksplice aborted the update because the symbols exported +by the kernel did not match Ksplice's expectations based on the ksplice_export +structures provided to Ksplice. + +already_reversed (P->A): Ksplice aborted the update because once an update has +been reversed, it cannot be applied again (without first being cleaned up and +reinitialized). + +module_busy (A->R): Ksplice aborted the undo operation because the target +Ksplice update is in use by another kernel module; specifically, either the +target Ksplice update exports a symbol that is in use by another module or +another Ksplice update depends on this Ksplice update. + +out_of_memory (P->A, A->R): Ksplice aborted the operation because a call to +kmalloc or vmalloc failed. + +code_busy (P->A, A->R): Ksplice aborted the operation because Ksplice was +unable to find a moment when one or more of the to-be-patched functions was not +a thread's kernel stack. + +unexpected_running_task (P->A, A->R): Ksplice aborted the operation because +Ksplice observed a running task during the kernel stack check, at a time when +Ksplice expected all tasks to be stopped by stop_machine. + +unexpected (P->A, A->R): Ksplice aborted the operation because it encountered +an unspecified internal error. This condition can only be caused by an invalid +input to Ksplice or a bug in Ksplice. + +D. /sys/kernel/ksplice/$kid/conflicts (mode 0400) + +This file is empty until Ksplice aborts an operation because of a code_busy +condition (see "abort_code" above). This conflicts file then contains +information about the process(es) that caused the stack check failure. + +Specifically, each line of this file consists of three space-separated values, +describing a single conflict: + +$program_name $program_pid $conflict_label + +$program_name is the name of the program with the conflict. +$program_pid is the pid of the program with the conflict. +$conflict_label is the Ksplice label of the function with the conflict. + +7. debugfs interface +-------------------- + +Ksplice exports a single file to debugfs for each Ksplice update. The file has +a name of the form ksplice_KID, where KID is the unique identifier of the +Ksplice update. It contains debugging information in a human-readable format. -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 7/9] Ksplice: x86-32 and x86-64 support 2008-09-13 4:49 ` [RFC PATCH 6/9] Ksplice documentation Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 8/9] Ksplice: ARM support Jeffrey Brian Arnold 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko Add support for Ksplice on x86. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> --- arch/x86/Kconfig | 1 + arch/x86/kernel/ksplice-arch.c | 91 ++++++++++++++++++++++++++++++++++++++++ arch/x86/mm/init_32.c | 6 +- arch/x86/mm/init_64.c | 4 +- 4 files changed, 97 insertions(+), 5 deletions(-) create mode 100644 arch/x86/kernel/ksplice-arch.c diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ed92864..b5ec2f4 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -23,6 +23,7 @@ config X86 select HAVE_OPROFILE select HAVE_IOREMAP_PROT select HAVE_KPROBES + select HAVE_KSPLICE select ARCH_WANT_OPTIONAL_GPIOLIB select HAVE_KRETPROBES select HAVE_DYNAMIC_FTRACE diff --git a/arch/x86/kernel/ksplice-arch.c b/arch/x86/kernel/ksplice-arch.c new file mode 100644 index 0000000..52314c4 --- /dev/null +++ b/arch/x86/kernel/ksplice-arch.c @@ -0,0 +1,91 @@ +/* Copyright (C) 2007-2008 Jeffrey Brian Arnold <jbarnold@mit.edu> + * Copyright (C) 2008 Anders Kaseorg <andersk@mit.edu>, + * Tim Abbott <tabbott@mit.edu> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +#define KSPLICE_IP(x) ((x)->thread.ip) +#define KSPLICE_SP(x) ((x)->thread.sp) + +static const struct ksplice_symbol trampoline_symbol = { + .name = NULL, + .label = "<trampoline>", +}; + +static const struct ksplice_reloc trampoline_reloc = { + .symbol = &trampoline_symbol, + .pcrel = 1, + .addend = -4, + .size = 4, + .dst_mask = 0xffffffffL, + .rightshift = 0, + .signed_addend = 1, +}; + +static abort_t trampoline_target(struct ksplice_pack *pack, unsigned long addr, + unsigned long *new_addr) +{ + abort_t ret; + unsigned char byte; + + if (probe_kernel_read(&byte, (void *)addr, sizeof(byte)) == -EFAULT) + return NO_MATCH; + + if (byte != 0xe9) + return NO_MATCH; + + ret = read_reloc_value(pack, &trampoline_reloc, addr + 1, new_addr); + if (ret != OK) + return ret; + + *new_addr += addr + 1; + return OK; +} + +static abort_t prepare_trampoline(struct ksplice_pack *pack, + struct ksplice_trampoline *t) +{ + t->size = 5; + t->trampoline[0] = 0xe9; + return write_reloc_value(pack, &trampoline_reloc, + (unsigned long)t->trampoline + 1, + t->repladdr - (t->oldaddr + 1)); +} + +static abort_t handle_paravirt(struct ksplice_pack *pack, + unsigned long pre_addr, unsigned long run_addr, + int *matched) +{ + unsigned char run[5], pre[5]; + *matched = 0; + + if (probe_kernel_read(&run, (void *)run_addr, sizeof(run)) == -EFAULT || + probe_kernel_read(&pre, (void *)pre_addr, sizeof(pre)) == -EFAULT) + return OK; + + if ((run[0] == 0xe8 && pre[0] == 0xe8) || + (run[0] == 0xe9 && pre[0] == 0xe9)) + if (run_addr + 1 + *(int32_t *)&run[1] == + pre_addr + 1 + *(int32_t *)&pre[1]) + *matched = 5; + return OK; +} + +static bool valid_stack_ptr(const struct thread_info *tinfo, const void *p) +{ + return p > (const void *)tinfo + && p <= (const void *)tinfo + THREAD_SIZE - sizeof(long); +} + diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c index d37f293..5998d5c 100644 --- a/arch/x86/mm/init_32.c +++ b/arch/x86/mm/init_32.c @@ -1036,8 +1036,8 @@ void mark_rodata_ro(void) unsigned long start = PFN_ALIGN(_text); unsigned long size = PFN_ALIGN(_etext) - start; -#ifndef CONFIG_DYNAMIC_FTRACE - /* Dynamic tracing modifies the kernel text section */ +#if !defined(CONFIG_DYNAMIC_FTRACE) && !defined(CONFIG_KSPLICE) + /* Dynamic tracing and Ksplice modify the kernel text section */ set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT); printk(KERN_INFO "Write protecting the kernel text: %luk\n", size >> 10); @@ -1050,7 +1050,7 @@ void mark_rodata_ro(void) printk(KERN_INFO "Testing CPA: write protecting again\n"); set_pages_ro(virt_to_page(start), size>>PAGE_SHIFT); #endif -#endif /* CONFIG_DYNAMIC_FTRACE */ +#endif /* !CONFIG_DYNAMIC_FTRACE && !CONFIG_KSPLICE */ start += size; size = (unsigned long)__end_rodata - start; diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index d3746ef..31db657 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -857,8 +857,8 @@ void mark_rodata_ro(void) unsigned long rodata_start = ((unsigned long)__start_rodata + PAGE_SIZE - 1) & PAGE_MASK; -#ifdef CONFIG_DYNAMIC_FTRACE - /* Dynamic tracing modifies the kernel text section */ +#if defined(CONFIG_DYNAMIC_FTRACE) || defined(CONFIG_KSPLICE) + /* Dynamic tracing and Ksplice modify the kernel text section */ start = rodata_start; #endif -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 8/9] Ksplice: ARM support 2008-09-13 4:49 ` [RFC PATCH 7/9] Ksplice: x86-32 and x86-64 support Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 9/9] Ksplice: Add support for building Ksplice as a module Jeffrey Brian Arnold 0 siblings, 1 reply; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko Add support for Ksplice on ARM. Signed-off-by: Jeffrey Brian Arnold <jbarnold@mit.edu> Signed-off-by: Anders Kaseorg <andersk@mit.edu> Signed-off-by: Tim Abbott <tabbott@mit.edu> --- arch/arm/Kconfig | 1 + arch/arm/kernel/ksplice-arch.c | 79 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 0 deletions(-) create mode 100644 arch/arm/kernel/ksplice-arch.c diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 70dba16..e73f205 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -14,6 +14,7 @@ config ARM select HAVE_OPROFILE select HAVE_ARCH_KGDB select HAVE_KPROBES if (!XIP_KERNEL) + select HAVE_KSPLICE if (!XIP_KERNEL) select HAVE_KRETPROBES if (HAVE_KPROBES) select HAVE_FTRACE if (!XIP_KERNEL) select HAVE_DYNAMIC_FTRACE if (HAVE_FTRACE) diff --git a/arch/arm/kernel/ksplice-arch.c b/arch/arm/kernel/ksplice-arch.c new file mode 100644 index 0000000..981cf75 --- /dev/null +++ b/arch/arm/kernel/ksplice-arch.c @@ -0,0 +1,79 @@ +/* Copyright (C) 2007-2008 Jeffrey Brian Arnold <jbarnold@mit.edu> + * Copyright (C) 2008 Anders Kaseorg <andersk@mit.edu>, + * Tim Abbott <tabbott@mit.edu> + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA + * 02110-1301, USA. + */ + +#include <linux/thread_info.h> +#define KSPLICE_IP(x) thread_saved_pc(x) +#define KSPLICE_SP(x) thread_saved_fp(x) + +static const struct ksplice_symbol trampoline_symbol = { + .name = NULL, + .label = "<trampoline>", +}; + +static const struct ksplice_reloc trampoline_reloc = { + .symbol = &trampoline_symbol, + .pcrel = 1, + .addend = -8, + .size = 4, + .dst_mask = 0x00ffffffL, + .rightshift = 2, + .signed_addend = 1, +}; + +static abort_t trampoline_target(struct ksplice_pack *pack, unsigned long addr, + unsigned long *new_addr) +{ + abort_t ret; + uint32_t word; + if (probe_kernel_read(&word, (void *)addr, sizeof(word)) == -EFAULT) + return NO_MATCH; + + if ((word & 0xff000000) != 0xea000000) + return NO_MATCH; + + ret = read_reloc_value(pack, &trampoline_reloc, addr, new_addr); + if (ret != OK) + return ret; + + *new_addr += addr; + return OK; +} + +static abort_t prepare_trampoline(struct ksplice_pack *pack, + struct ksplice_trampoline *t) +{ + t->size = 4; + *(uint32_t *)t->trampoline = 0xea000000; + return write_reloc_value(pack, &trampoline_reloc, + (unsigned long)t->trampoline, + t->repladdr - t->oldaddr); +} + +static abort_t handle_paravirt(struct ksplice_pack *pack, + unsigned long pre_addr, unsigned long run_addr, + int *matched) +{ + *matched = 0; + return OK; +} + +static bool valid_stack_ptr(const struct thread_info *tinfo, const void *p) +{ + return p > (const void *)tinfo + && p <= (const void *)tinfo + THREAD_SIZE - sizeof(long); +} -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* [RFC PATCH 9/9] Ksplice: Add support for building Ksplice as a module 2008-09-13 4:49 ` [RFC PATCH 8/9] Ksplice: ARM support Jeffrey Brian Arnold @ 2008-09-13 4:49 ` Jeffrey Brian Arnold 0 siblings, 0 replies; 14+ messages in thread From: Jeffrey Brian Arnold @ 2008-09-13 4:49 UTC (permalink / raw) To: linux-kernel; +Cc: Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko From: Tim Abbott <tabbott@mit.edu> Export the symbols needed to build Ksplice as a module. Signed-off-by: Tim Abbott <tabbott@mit.edu> --- arch/Kconfig | 2 +- arch/x86/mm/init_32.c | 5 +++-- arch/x86/mm/init_64.c | 3 ++- kernel/kallsyms.c | 1 + kernel/module.c | 9 +++++++++ kernel/sched.c | 1 + 6 files changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 9d5a843..cfbbcf3 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -28,7 +28,7 @@ config KPROBES If in doubt, say "N". config KSPLICE - bool "Ksplice rebootless kernel updates" + tristate "Ksplice rebootless kernel updates" depends on KALLSYMS_ALL && MODULE_UNLOAD && SYSFS && \ FUNCTION_DATA_SECTIONS depends on HAVE_KSPLICE diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c index 5998d5c..549b021 100644 --- a/arch/x86/mm/init_32.c +++ b/arch/x86/mm/init_32.c @@ -1036,7 +1036,8 @@ void mark_rodata_ro(void) unsigned long start = PFN_ALIGN(_text); unsigned long size = PFN_ALIGN(_etext) - start; -#if !defined(CONFIG_DYNAMIC_FTRACE) && !defined(CONFIG_KSPLICE) +#if !defined(CONFIG_DYNAMIC_FTRACE) && \ + !defined(CONFIG_KSPLICE) && !defined(CONFIG_KSPLICE_MODULE) /* Dynamic tracing and Ksplice modify the kernel text section */ set_pages_ro(virt_to_page(start), size >> PAGE_SHIFT); printk(KERN_INFO "Write protecting the kernel text: %luk\n", @@ -1050,7 +1051,7 @@ void mark_rodata_ro(void) printk(KERN_INFO "Testing CPA: write protecting again\n"); set_pages_ro(virt_to_page(start), size>>PAGE_SHIFT); #endif -#endif /* !CONFIG_DYNAMIC_FTRACE && !CONFIG_KSPLICE */ +#endif /* !CONFIG_DYNAMIC_FTRACE && !CONFIG_KSPLICE && !CONFIG_KSPLICE_MODULE */ start += size; size = (unsigned long)__end_rodata - start; diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 31db657..dec6325 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -857,7 +857,8 @@ void mark_rodata_ro(void) unsigned long rodata_start = ((unsigned long)__start_rodata + PAGE_SIZE - 1) & PAGE_MASK; -#if defined(CONFIG_DYNAMIC_FTRACE) || defined(CONFIG_KSPLICE) +#if defined(CONFIG_DYNAMIC_FTRACE) || \ + defined(CONFIG_KSPLICE) || defined(CONFIG_KSPLICE_MODULE) /* Dynamic tracing and Ksplice modify the kernel text section */ start = rodata_start; #endif diff --git a/kernel/kallsyms.c b/kernel/kallsyms.c index 47d3e94..ef47021 100644 --- a/kernel/kallsyms.c +++ b/kernel/kallsyms.c @@ -178,6 +178,7 @@ int kallsyms_on_each_symbol(int (*fn)(void *, const char *, struct module *, } return module_kallsyms_on_each_symbol(fn, data); } +EXPORT_SYMBOL_GPL(kallsyms_on_each_symbol); static unsigned long get_symbol_pos(unsigned long addr, unsigned long *symbolsize, diff --git a/kernel/module.c b/kernel/module.c index 2acb646..d2fa540 100644 --- a/kernel/module.c +++ b/kernel/module.c @@ -63,6 +63,7 @@ /* List of modules, protected by module_mutex or preempt_disable * (add/delete uses stop_machine). */ DEFINE_MUTEX(module_mutex); +EXPORT_SYMBOL_GPL(module_mutex); static LIST_HEAD(modules); /* Waiting for a module to finish initializing? */ @@ -324,6 +325,7 @@ const struct kernel_symbol *find_symbol(const char *name, DEBUGP("Failed to find symbol %s\n", name); return NULL; } +EXPORT_SYMBOL_GPL(find_symbol); /* Search for module by name: must hold module_mutex. */ struct module *find_module(const char *name) @@ -336,6 +338,7 @@ struct module *find_module(const char *name) } return NULL; } +EXPORT_SYMBOL_GPL(find_module); #ifdef CONFIG_SMP /* Number of blocks used and allocated. */ @@ -612,6 +615,7 @@ int use_module(struct module *a, struct module *b) no_warn = sysfs_create_link(b->holders_dir, &a->mkobj.kobj, a->name); return 1; } +EXPORT_SYMBOL_GPL(use_module); /* Clear the unload stuff of the module. */ static void module_unload_free(struct module *mod) @@ -878,6 +882,7 @@ int use_module(struct module *a, struct module *b) { return strong_try_module_get(b) == 0; } +EXPORT_SYMBOL_GPL(use_module); static inline void module_unload_init(struct module *mod) { @@ -2689,6 +2694,7 @@ struct module *__module_text_address(unsigned long addr) return mod; return NULL; } +EXPORT_SYMBOL_GPL(__module_text_address); struct module *module_text_address(unsigned long addr) { @@ -2700,6 +2706,7 @@ struct module *module_text_address(unsigned long addr) return mod; } +EXPORT_SYMBOL_GPL(module_text_address); struct module *__module_data_address(unsigned long addr) { @@ -2717,6 +2724,7 @@ struct module *__module_data_address(unsigned long addr) } return NULL; } +EXPORT_SYMBOL_GPL(__module_data_address); struct module *module_data_address(unsigned long addr) { @@ -2728,6 +2736,7 @@ struct module *module_data_address(unsigned long addr) return mod; } +EXPORT_SYMBOL_GPL(module_data_address); /* Don't grab lock, we're oopsing. */ void print_modules(void) diff --git a/kernel/sched.c b/kernel/sched.c index cc1f81b..bd385b6 100644 --- a/kernel/sched.c +++ b/kernel/sched.c @@ -1736,6 +1736,7 @@ inline int task_curr(const struct task_struct *p) { return cpu_curr(task_cpu(p)) == p; } +EXPORT_SYMBOL_GPL(task_curr); static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu) { -- 1.5.4.3 ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH 0/9] Ksplice: Rebootless kernel updates 2008-09-13 4:49 [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Jeffrey Brian Arnold 2008-09-13 4:49 ` [RFC PATCH 1/9] Ksplice: Make find_symbol return a struct kernel_symbol Jeffrey Brian Arnold @ 2008-09-13 5:12 ` Alexey Dobriyan 2008-09-14 20:24 ` Jeff Arnold 1 sibling, 1 reply; 14+ messages in thread From: Alexey Dobriyan @ 2008-09-13 5:12 UTC (permalink / raw) To: Jeffrey Brian Arnold Cc: linux-kernel, Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko On Sat, Sep 13, 2008 at 12:49:46AM -0400, Jeffrey Brian Arnold wrote: > These RFC patches add support for Ksplice [1], a rebootless update system, > to the kernel. > The basic idea is that Ksplice can apply an update to a running > Linux kernel based only on a source code patch and the original kernel > source. Please, taint kernel after first update so some of us can ignore oopses of hopelessly broken kernels. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [RFC PATCH 0/9] Ksplice: Rebootless kernel updates 2008-09-13 5:12 ` [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Alexey Dobriyan @ 2008-09-14 20:24 ` Jeff Arnold 0 siblings, 0 replies; 14+ messages in thread From: Jeff Arnold @ 2008-09-14 20:24 UTC (permalink / raw) To: Alexey Dobriyan Cc: linux-kernel, Tim Abbott, Anders Kaseorg, Waseem Daher, Denys Vlasenko On Sat, 13 Sep 2008, Alexey Dobriyan wrote: > Please, taint kernel after first update so some of us can ignore oopses > of hopelessly broken kernels. OK. Jeff Arnold jbarnold@mit.edu ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2008-09-14 20:24 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2008-09-13 4:49 [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 1/9] Ksplice: Make find_symbol return a struct kernel_symbol Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 2/9] Ksplice: Add module_data_address (the analogue of module_text_address) Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 3/9] Ksplice: Add functions for walking kallsyms symbols Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 4/9] Ksplice: Make needed symbol and module functions global Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 5/9] Ksplice: Architecture-independent core Jeffrey Brian Arnold
[not found] ` <8d86b7714a1012014ce3533aa24dc88704fe14a6.1221271644.git.jbarnold@mit.edu>
2008-09-13 4:49 ` [RFC PATCH 5/9] Ksplice core (architecture independent) Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 6/9] Ksplice: Documentation Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 6/9] Ksplice documentation Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 7/9] Ksplice: x86-32 and x86-64 support Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 8/9] Ksplice: ARM support Jeffrey Brian Arnold
2008-09-13 4:49 ` [RFC PATCH 9/9] Ksplice: Add support for building Ksplice as a module Jeffrey Brian Arnold
2008-09-13 5:12 ` [RFC PATCH 0/9] Ksplice: Rebootless kernel updates Alexey Dobriyan
2008-09-14 20:24 ` Jeff Arnold
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®