* Re: [PATCH net-next 8/9] net: filter: rework/optimize internal BPF interpreter's instruction set
2014-03-21 12:20 ` [PATCH net-next 8/9] net: filter: rework/optimize internal BPF interpreter's instruction set Daniel Borkmann
@ 2014-03-21 15:40 ` Kees Cook
0 siblings, 0 replies; 2+ messages in thread
From: Kees Cook @ 2014-03-21 15:40 UTC (permalink / raw)
To: Daniel Borkmann
Cc: David S. Miller, Alexei Starovoitov, netdev, Hagen Paul Pfeifer,
Paul Moore, Ingo Molnar, H. Peter Anvin, LKML, Will Drewry
On Fri, Mar 21, 2014 at 6:20 AM, Daniel Borkmann <dborkman@redhat.com> wrote:
> From: Alexei Starovoitov <ast@plumgrid.com>
>
> This patch replaces/reworks the kernel-internel BPF interpreter with
> an optimized BPF instruction set format that is modelled closer to
> mimic native instruction sets and is designed to be JITed with one to
> one mapping. Thus, the new interpreter is noticeably faster than the
> current implementation of sk_run_filter(); mainly for two reasons:
>
> 1. Fall-through jumps:
>
> BPF jump instructions are forced to go either 'true' or 'false'
> branch which causes branch-miss penalty. The new BPF jump
> instructions have only one branch and fall-through otherwise,
> which fits the CPU branch predictor logic better. `perf stat`
> shows drastic difference for branch-misses between the old and
> new code.
>
> 2. Jump-threaded implementation of interpreter vs switch
> statement:
>
> Instead of single tablejump at the top of 'switch' statement,
> gcc will now generate multiple tablejump instructions, which
> helps CPU branch predictor logic.
>
> In short, the internal format extends BPF in the following way (more
> details can be taken from the appended documentation):
>
> - Number of registers increase from 2 to 10
> - Register width increases from 32-bit to 64-bit
> - Conditional jt/jf targets replaced with jt/fall-through,
> and forward/backward jumps now possible as well
> - Adds signed > and >= insns
> - 16 4-byte stack slots for register spill-fill replaced
> with up to 512 bytes of multi-use stack space
> - Introduction of bpf_call insn and register passing convention
> for zero overhead calls from/to other kernel functions
> - Adds arithmetic right shift insn
> - Adds swab insns for 32/64-bit
> - Adds atomic_add insn
> - Old tax/txa insns are replaced with 'mov dst,src' insn
>
> Note that the verification of filters is still being done through
> sk_chk_filter(), so filters from user- or kernel space are verified
> in the same way as we do now. We reuse current BPF JIT compilers
> in a way that this upgrade would even be fine as is, but nevertheless
> allows for a successive upgrade of BPF JIT compilers to the new
> format. The internal instruction set migration is being done after
> the probing for JIT compilation, so in case JIT compilers are able
> to create a native opcode image, we're going to use that, and in all
> other cases we're doing a follow-up migration of the BPG program's
> instruction set, so that it can be transparently run in the new
> interpreter.
>
> Performance of two BPF filters generated by libpcap resp. bpf_asm
> was measured on x86_64, i386 and arm32 (other libpcap programs
> have similar performance differences):
>
> fprog #1 is taken from Documentation/networking/filter.txt:
> tcpdump -i eth0 port 22 -dd
>
> fprog #2 is taken from 'man tcpdump':
> tcpdump -i eth0 'tcp port 22 and (((ip[2:2] - ((ip[0]&0xf)<<2)) -
> ((tcp[12]&0xf0)>>2)) != 0)' -dd
>
> Raw performance data from BPF micro-benchmark: SK_RUN_FILTER on the
> same SKB (cache-hit) or 10k SKBs (cache-miss); time in nsec per
> call, smaller is better:
>
> --x86_64--
> fprog #1 fprog #1 fprog #2 fprog #2
> cache-hit cache-miss cache-hit cache-miss
> old BPF 90 101 192 202
> new BPF 31 71 47 97
> old BPF jit 12 34 17 44
> new BPF jit TBD
>
> --i386--
> fprog #1 fprog #1 fprog #2 fprog #2
> cache-hit cache-miss cache-hit cache-miss
> old BPF 107 136 227 252
> new BPF 40 119 69 172
>
> --arm32--
> fprog #1 fprog #1 fprog #2 fprog #2
> cache-hit cache-miss cache-hit cache-miss
> old BPF 202 300 475 540
> new BPF 180 270 330 470
> old BPF jit 26 182 37 202
> new BPF jit TBD
>
> Thus, without changing any userland BPF filters, applications on
> top of AF_PACKET (or other families) such as libpcap/tcpdump, cls_bpf
> classifier, netfilter's xt_bpf, team driver's load-balancing mode,
> and many more will have better interpreter filtering performance.
>
> While we are replacing the internal BPF interpreter, we also need
> to convert seccomp BPF in the same step to make use of the new
> internal structure since it makes use of lower-level API details
> without being further decoupled through higher-level calls like
> sk_unattached_filter_{create,destroy}(), for example.
>
> Just as for normal socket filtering, also seccomp BPF experiences
> a time-to-verdict speedup:
>
> 05-sim-long_jumps.c of libseccomp was used as micro-benchmark:
>
> seccomp_rule_add_exact(ctx,...
> seccomp_rule_add_exact(ctx,...
>
> rc = seccomp_load(ctx);
>
> for (i = 0; i < 10000000; i++)
> syscall(199, 100);
>
> 'short filter' has 2 rules
> 'large filter' has 200 rules
>
> 'short filter' performance is slightly better on x86_64/i386/arm32
> 'large filter' is much faster on x86_64 and i386 and shows no
> difference on arm32
>
> --x86_64-- short filter
> old BPF: 2.7 sec
> 39.12% bench libc-2.15.so [.] syscall
> 8.10% bench [kernel.kallsyms] [k] sk_run_filter
> 6.31% bench [kernel.kallsyms] [k] system_call
> 5.59% bench [kernel.kallsyms] [k] trace_hardirqs_on_caller
> 4.37% bench [kernel.kallsyms] [k] trace_hardirqs_off_caller
> 3.70% bench [kernel.kallsyms] [k] __secure_computing
> 3.67% bench [kernel.kallsyms] [k] lock_is_held
> 3.03% bench [kernel.kallsyms] [k] seccomp_bpf_load
> new BPF: 2.58 sec
> 42.05% bench libc-2.15.so [.] syscall
> 6.91% bench [kernel.kallsyms] [k] system_call
> 6.25% bench [kernel.kallsyms] [k] trace_hardirqs_on_caller
> 6.07% bench [kernel.kallsyms] [k] __secure_computing
> 5.08% bench [kernel.kallsyms] [k] sk_run_filter_int_seccomp
>
> --arm32-- short filter
> old BPF: 4.0 sec
> 39.92% bench [kernel.kallsyms] [k] vector_swi
> 16.60% bench [kernel.kallsyms] [k] sk_run_filter
> 14.66% bench libc-2.17.so [.] syscall
> 5.42% bench [kernel.kallsyms] [k] seccomp_bpf_load
> 5.10% bench [kernel.kallsyms] [k] __secure_computing
> new BPF: 3.7 sec
> 35.93% bench [kernel.kallsyms] [k] vector_swi
> 21.89% bench libc-2.17.so [.] syscall
> 13.45% bench [kernel.kallsyms] [k] sk_run_filter_int_seccomp
> 6.25% bench [kernel.kallsyms] [k] __secure_computing
> 3.96% bench [kernel.kallsyms] [k] syscall_trace_exit
>
> --x86_64-- large filter
> old BPF: 8.6 seconds
> 73.38% bench [kernel.kallsyms] [k] sk_run_filter
> 10.70% bench libc-2.15.so [.] syscall
> 5.09% bench [kernel.kallsyms] [k] seccomp_bpf_load
> 1.97% bench [kernel.kallsyms] [k] system_call
> new BPF: 5.7 seconds
> 66.20% bench [kernel.kallsyms] [k] sk_run_filter_int_seccomp
> 16.75% bench libc-2.15.so [.] syscall
> 3.31% bench [kernel.kallsyms] [k] system_call
> 2.88% bench [kernel.kallsyms] [k] __secure_computing
>
> --i386-- large filter
> old BPF: 5.4 sec
> new BPF: 3.8 sec
>
> --arm32-- large filter
> old BPF: 13.5 sec
> 73.88% bench [kernel.kallsyms] [k] sk_run_filter
> 10.29% bench [kernel.kallsyms] [k] vector_swi
> 6.46% bench libc-2.17.so [.] syscall
> 2.94% bench [kernel.kallsyms] [k] seccomp_bpf_load
> 1.19% bench [kernel.kallsyms] [k] __secure_computing
> 0.87% bench [kernel.kallsyms] [k] sys_getuid
> new BPF: 13.5 sec
> 76.08% bench [kernel.kallsyms] [k] sk_run_filter_int_seccomp
> 10.98% bench [kernel.kallsyms] [k] vector_swi
> 5.87% bench libc-2.17.so [.] syscall
> 1.77% bench [kernel.kallsyms] [k] __secure_computing
> 0.93% bench [kernel.kallsyms] [k] sys_getuid
>
> BPF filters generated by seccomp are very branchy, so the new
> internal BPF performance is better than the old one. Performance
> gains will be even higher when BPF JIT is committed for the
> new structure, which is planned in future work (as successive
> JIT migrations).
>
> BPF has also been stress-tested with trinity's BPF fuzzer.
>
> Joint work with Daniel Borkmann.
>
> References: http://thread.gmane.org/gmane.linux.kernel/1665858
> Signed-off-by: Alexei Starovoitov <ast@plumgrid.com>
> Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
> Cc: Hagen Paul Pfeifer <hagen@jauu.net>
> Cc: Kees Cook <keescook@chromium.org>
> Cc: Paul Moore <pmoore@redhat.com>
> Cc: Ingo Molnar <mingo@kernel.org>
> Cc: H. Peter Anvin <hpa@linux.intel.com>
> Cc: linux-kernel@vger.kernel.org
This looks great, thanks for all the seccomp testing!
Acked-by: Kees Cook <keescook@chromium.org>
-Kees
> ---
> v1 -> v10 history at:
> - http://thread.gmane.org/gmane.linux.kernel/1665858
>
> include/linux/filter.h | 66 ++-
> include/linux/seccomp.h | 1 -
> kernel/seccomp.c | 119 ++--
> net/core/filter.c | 1415 +++++++++++++++++++++++++++++++++++++----------
> 4 files changed, 1229 insertions(+), 372 deletions(-)
>
> diff --git a/include/linux/filter.h b/include/linux/filter.h
> index 9bde3ed..3ea12fa 100644
> --- a/include/linux/filter.h
> +++ b/include/linux/filter.h
> @@ -9,13 +9,50 @@
> #include <linux/workqueue.h>
> #include <uapi/linux/filter.h>
>
> -#ifdef CONFIG_COMPAT
> -/*
> - * A struct sock_filter is architecture independent.
> +/* Internally used and optimized filter representation with extended
> + * instruction set based on top of classic BPF.
> */
> +
> +/* instruction classes */
> +#define BPF_ALU64 0x07 /* alu mode in double word width */
> +
> +/* ld/ldx fields */
> +#define BPF_DW 0x18 /* double word */
> +#define BPF_XADD 0xc0 /* exclusive add */
> +
> +/* alu/jmp fields */
> +#define BPF_MOV 0xb0 /* mov reg to reg */
> +#define BPF_ARSH 0xc0 /* sign extending arithmetic shift right */
> +#define BPF_BSWAP 0xd0 /* swap 4 or 8 bytes of 64-bit register */
> +
> +#define BPF_JNE 0x50 /* jump != */
> +#define BPF_JSGT 0x60 /* SGT is signed '>', GT in x86 */
> +#define BPF_JSGE 0x70 /* SGE is signed '>=', GE in x86 */
> +#define BPF_CALL 0x80 /* function call */
> +
> +/* BPF has 10 general purpose 64-bit registers and stack frame. */
> +#define MAX_BPF_REG 11
> +
> +/* BPF program can access up to 512 bytes of stack space. */
> +#define MAX_BPF_STACK 512
> +
> +/* Context and stack frame pointer register positions. */
> +#define CTX_REG 1
> +#define FP_REG 10
> +
> +struct sock_filter_int {
> + __u8 code; /* opcode */
> + __u8 a_reg:4; /* dest register */
> + __u8 x_reg:4; /* source register */
> + __s16 off; /* signed offset */
> + __s32 imm; /* signed immediate constant */
> +};
> +
> +#ifdef CONFIG_COMPAT
> +/* A struct sock_filter is architecture independent. */
> struct compat_sock_fprog {
> u16 len;
> - compat_uptr_t filter; /* struct sock_filter * */
> + compat_uptr_t filter; /* struct sock_filter * */
> };
> #endif
>
> @@ -26,6 +63,7 @@ struct sock_fprog_kern {
>
> struct sk_buff;
> struct sock;
> +struct seccomp_data;
>
> struct sk_filter {
> atomic_t refcnt;
> @@ -34,9 +72,10 @@ struct sk_filter {
> struct sock_fprog_kern *orig_prog; /* Original BPF program */
> struct rcu_head rcu;
> unsigned int (*bpf_func)(const struct sk_buff *skb,
> - const struct sock_filter *filter);
> + const struct sock_filter_int *filter);
> union {
> - struct sock_filter insns[0];
> + struct sock_filter insns[0];
> + struct sock_filter_int insnsi[0];
> struct work_struct work;
> };
> };
> @@ -50,9 +89,18 @@ static inline unsigned int sk_filter_size(unsigned int proglen)
> #define sk_filter_proglen(fprog) \
> (fprog->len * sizeof(fprog->filter[0]))
>
> +#define SK_RUN_FILTER(filter, ctx) \
> + (*filter->bpf_func)(ctx, filter->insnsi)
> +
> int sk_filter(struct sock *sk, struct sk_buff *skb);
> -unsigned int sk_run_filter(const struct sk_buff *skb,
> - const struct sock_filter *filter);
> +
> +u32 sk_run_filter_int_seccomp(const struct seccomp_data *ctx,
> + const struct sock_filter_int *insni);
> +u32 sk_run_filter_int_skb(const struct sk_buff *ctx,
> + const struct sock_filter_int *insni);
> +
> +int sk_convert_filter(struct sock_filter *prog, int len,
> + struct sock_filter_int *new_prog, int *new_len);
>
> int sk_unattached_filter_create(struct sk_filter **pfp,
> struct sock_fprog *fprog);
> @@ -86,7 +134,6 @@ static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen,
> print_hex_dump(KERN_ERR, "JIT code: ", DUMP_PREFIX_OFFSET,
> 16, 1, image, proglen, false);
> }
> -#define SK_RUN_FILTER(FILTER, SKB) (*FILTER->bpf_func)(SKB, FILTER->insns)
> #else
> #include <linux/slab.h>
> static inline void bpf_jit_compile(struct sk_filter *fp)
> @@ -96,7 +143,6 @@ static inline void bpf_jit_free(struct sk_filter *fp)
> {
> kfree(fp);
> }
> -#define SK_RUN_FILTER(FILTER, SKB) sk_run_filter(SKB, FILTER->insns)
> #endif
>
> static inline int bpf_tell_extensions(void)
> diff --git a/include/linux/seccomp.h b/include/linux/seccomp.h
> index 6f19cfd..4054b09 100644
> --- a/include/linux/seccomp.h
> +++ b/include/linux/seccomp.h
> @@ -76,7 +76,6 @@ static inline int seccomp_mode(struct seccomp *s)
> #ifdef CONFIG_SECCOMP_FILTER
> extern void put_seccomp_filter(struct task_struct *tsk);
> extern void get_seccomp_filter(struct task_struct *tsk);
> -extern u32 seccomp_bpf_load(int off);
> #else /* CONFIG_SECCOMP_FILTER */
> static inline void put_seccomp_filter(struct task_struct *tsk)
> {
> diff --git a/kernel/seccomp.c b/kernel/seccomp.c
> index b7a1004..4f18e75 100644
> --- a/kernel/seccomp.c
> +++ b/kernel/seccomp.c
> @@ -55,60 +55,33 @@ struct seccomp_filter {
> atomic_t usage;
> struct seccomp_filter *prev;
> unsigned short len; /* Instruction count */
> - struct sock_filter insns[];
> + struct sock_filter_int insnsi[];
> };
>
> /* Limit any path through the tree to 256KB worth of instructions. */
> #define MAX_INSNS_PER_PATH ((1 << 18) / sizeof(struct sock_filter))
>
> -/**
> - * get_u32 - returns a u32 offset into data
> - * @data: a unsigned 64 bit value
> - * @index: 0 or 1 to return the first or second 32-bits
> - *
> - * This inline exists to hide the length of unsigned long. If a 32-bit
> - * unsigned long is passed in, it will be extended and the top 32-bits will be
> - * 0. If it is a 64-bit unsigned long, then whatever data is resident will be
> - * properly returned.
> - *
> +/*
> * Endianness is explicitly ignored and left for BPF program authors to manage
> * as per the specific architecture.
> */
> -static inline u32 get_u32(u64 data, int index)
> +static void populate_seccomp_data(struct seccomp_data *sd)
> {
> - return ((u32 *)&data)[index];
> -}
> + struct task_struct *task = current;
> + struct pt_regs *regs = task_pt_regs(task);
>
> -/* Helper for bpf_load below. */
> -#define BPF_DATA(_name) offsetof(struct seccomp_data, _name)
> -/**
> - * bpf_load: checks and returns a pointer to the requested offset
> - * @off: offset into struct seccomp_data to load from
> - *
> - * Returns the requested 32-bits of data.
> - * seccomp_check_filter() should assure that @off is 32-bit aligned
> - * and not out of bounds. Failure to do so is a BUG.
> - */
> -u32 seccomp_bpf_load(int off)
> -{
> - struct pt_regs *regs = task_pt_regs(current);
> - if (off == BPF_DATA(nr))
> - return syscall_get_nr(current, regs);
> - if (off == BPF_DATA(arch))
> - return syscall_get_arch(current, regs);
> - if (off >= BPF_DATA(args[0]) && off < BPF_DATA(args[6])) {
> - unsigned long value;
> - int arg = (off - BPF_DATA(args[0])) / sizeof(u64);
> - int index = !!(off % sizeof(u64));
> - syscall_get_arguments(current, regs, arg, 1, &value);
> - return get_u32(value, index);
> - }
> - if (off == BPF_DATA(instruction_pointer))
> - return get_u32(KSTK_EIP(current), 0);
> - if (off == BPF_DATA(instruction_pointer) + sizeof(u32))
> - return get_u32(KSTK_EIP(current), 1);
> - /* seccomp_check_filter should make this impossible. */
> - BUG();
> + sd->nr = syscall_get_nr(task, regs);
> + sd->arch = syscall_get_arch(task, regs);
> +
> + /* Unroll syscall_get_args to help gcc on arm. */
> + syscall_get_arguments(task, regs, 0, 1, (unsigned long *) &sd->args[0]);
> + syscall_get_arguments(task, regs, 1, 1, (unsigned long *) &sd->args[1]);
> + syscall_get_arguments(task, regs, 2, 1, (unsigned long *) &sd->args[2]);
> + syscall_get_arguments(task, regs, 3, 1, (unsigned long *) &sd->args[3]);
> + syscall_get_arguments(task, regs, 4, 1, (unsigned long *) &sd->args[4]);
> + syscall_get_arguments(task, regs, 5, 1, (unsigned long *) &sd->args[5]);
> +
> + sd->instruction_pointer = KSTK_EIP(task);
> }
>
> /**
> @@ -133,17 +106,17 @@ static int seccomp_check_filter(struct sock_filter *filter, unsigned int flen)
>
> switch (code) {
> case BPF_S_LD_W_ABS:
> - ftest->code = BPF_S_ANC_SECCOMP_LD_W;
> + ftest->code = BPF_LDX | BPF_W | BPF_ABS;
> /* 32-bit aligned and not out of bounds. */
> if (k >= sizeof(struct seccomp_data) || k & 3)
> return -EINVAL;
> continue;
> case BPF_S_LD_W_LEN:
> - ftest->code = BPF_S_LD_IMM;
> + ftest->code = BPF_LD | BPF_IMM;
> ftest->k = sizeof(struct seccomp_data);
> continue;
> case BPF_S_LDX_W_LEN:
> - ftest->code = BPF_S_LDX_IMM;
> + ftest->code = BPF_LDX | BPF_IMM;
> ftest->k = sizeof(struct seccomp_data);
> continue;
> /* Explicitly include allowed calls. */
> @@ -185,6 +158,7 @@ static int seccomp_check_filter(struct sock_filter *filter, unsigned int flen)
> case BPF_S_JMP_JGT_X:
> case BPF_S_JMP_JSET_K:
> case BPF_S_JMP_JSET_X:
> + sk_decode_filter(ftest, ftest);
> continue;
> default:
> return -EINVAL;
> @@ -202,18 +176,21 @@ static int seccomp_check_filter(struct sock_filter *filter, unsigned int flen)
> static u32 seccomp_run_filters(int syscall)
> {
> struct seccomp_filter *f;
> + struct seccomp_data sd;
> u32 ret = SECCOMP_RET_ALLOW;
>
> /* Ensure unexpected behavior doesn't result in failing open. */
> if (WARN_ON(current->seccomp.filter == NULL))
> return SECCOMP_RET_KILL;
>
> + populate_seccomp_data(&sd);
> +
> /*
> * All filters in the list are evaluated and the lowest BPF return
> * value always takes priority (ignoring the DATA).
> */
> for (f = current->seccomp.filter; f; f = f->prev) {
> - u32 cur_ret = sk_run_filter(NULL, f->insns);
> + u32 cur_ret = sk_run_filter_int_seccomp(&sd, f->insnsi);
> if ((cur_ret & SECCOMP_RET_ACTION) < (ret & SECCOMP_RET_ACTION))
> ret = cur_ret;
> }
> @@ -231,6 +208,8 @@ static long seccomp_attach_filter(struct sock_fprog *fprog)
> struct seccomp_filter *filter;
> unsigned long fp_size = fprog->len * sizeof(struct sock_filter);
> unsigned long total_insns = fprog->len;
> + struct sock_filter *fp;
> + int new_len;
> long ret;
>
> if (fprog->len == 0 || fprog->len > BPF_MAXINSNS)
> @@ -252,28 +231,43 @@ static long seccomp_attach_filter(struct sock_fprog *fprog)
> CAP_SYS_ADMIN) != 0)
> return -EACCES;
>
> - /* Allocate a new seccomp_filter */
> - filter = kzalloc(sizeof(struct seccomp_filter) + fp_size,
> - GFP_KERNEL|__GFP_NOWARN);
> - if (!filter)
> + fp = kzalloc(fp_size, GFP_KERNEL|__GFP_NOWARN);
> + if (!fp)
> return -ENOMEM;
> - atomic_set(&filter->usage, 1);
> - filter->len = fprog->len;
>
> /* Copy the instructions from fprog. */
> ret = -EFAULT;
> - if (copy_from_user(filter->insns, fprog->filter, fp_size))
> - goto fail;
> + if (copy_from_user(fp, fprog->filter, fp_size))
> + goto free_prog;
>
> /* Check and rewrite the fprog via the skb checker */
> - ret = sk_chk_filter(filter->insns, filter->len);
> + ret = sk_chk_filter(fp, fprog->len);
> if (ret)
> - goto fail;
> + goto free_prog;
>
> /* Check and rewrite the fprog for seccomp use */
> - ret = seccomp_check_filter(filter->insns, filter->len);
> + ret = seccomp_check_filter(fp, fprog->len);
> + if (ret)
> + goto free_prog;
> +
> + /* Convert 'sock_filter' insns to 'sock_filter_int' insns */
> + ret = sk_convert_filter(fp, fprog->len, NULL, &new_len);
> + if (ret)
> + goto free_prog;
> +
> + /* Allocate a new seccomp_filter */
> + filter = kzalloc(sizeof(struct seccomp_filter) +
> + sizeof(struct sock_filter_int) * new_len,
> + GFP_KERNEL|__GFP_NOWARN);
> + if (!filter)
> + goto free_prog;
> +
> + ret = sk_convert_filter(fp, fprog->len, filter->insnsi, &new_len);
> if (ret)
> - goto fail;
> + goto free_filter;
> +
> + atomic_set(&filter->usage, 1);
> + filter->len = new_len;
>
> /*
> * If there is an existing filter, make it the prev and don't drop its
> @@ -282,8 +276,11 @@ static long seccomp_attach_filter(struct sock_fprog *fprog)
> filter->prev = current->seccomp.filter;
> current->seccomp.filter = filter;
> return 0;
> -fail:
> +
> +free_filter:
> kfree(filter);
> +free_prog:
> + kfree(fp);
> return ret;
> }
>
> diff --git a/net/core/filter.c b/net/core/filter.c
> index 976edc6..683f1e8 100644
> --- a/net/core/filter.c
> +++ b/net/core/filter.c
> @@ -1,11 +1,16 @@
> /*
> * Linux Socket Filter - Kernel level socket filtering
> *
> - * Author:
> - * Jay Schulist <jschlst@samba.org>
> + * Based on the design of the Berkeley Packet Filter. The new
> + * internal format has been designed by PLUMgrid:
> *
> - * Based on the design of:
> - * - The Berkeley Packet Filter
> + * Copyright (c) 2011 - 2014 PLUMgrid, http://plumgrid.com
> + *
> + * Authors:
> + *
> + * Jay Schulist <jschlst@samba.org>
> + * Alexei Starovoitov <ast@plumgrid.com>
> + * Daniel Borkmann <dborkman@redhat.com>
> *
> * This program is free software; you can redistribute it and/or
> * modify it under the terms of the GNU General Public License
> @@ -35,6 +40,7 @@
> #include <linux/timer.h>
> #include <asm/uaccess.h>
> #include <asm/unaligned.h>
> +#include <asm/byteorder.h>
> #include <linux/filter.h>
> #include <linux/ratelimit.h>
> #include <linux/seccomp.h>
> @@ -108,304 +114,1002 @@ int sk_filter(struct sock *sk, struct sk_buff *skb)
> }
> EXPORT_SYMBOL(sk_filter);
>
> +/* Base function for offset calculation. Needs to go into .text section,
> + * therefore keeping it non-static as well; will also be used by JITs
> + * anyway later on, so do not let the compiler omit it.
> + */
> +noinline u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5)
> +{
> + return 0;
> +}
> +
> /**
> - * sk_run_filter - run a filter on a socket
> - * @skb: buffer to run the filter on
> + * __sk_run_filter - run a filter on a given context
> + * @ctx: buffer to run the filter on
> * @fentry: filter to apply
> *
> - * Decode and apply filter instructions to the skb->data.
> - * Return length to keep, 0 for none. @skb is the data we are
> - * filtering, @filter is the array of filter instructions.
> - * Because all jumps are guaranteed to be before last instruction,
> - * and last instruction guaranteed to be a RET, we dont need to check
> - * flen. (We used to pass to this function the length of filter)
> + * Decode and apply filter instructions to the skb->data. Return length to
> + * keep, 0 for none. @ctx is the data we are operating on, @filter is the
> + * array of filter instructions.
> */
> -unsigned int sk_run_filter(const struct sk_buff *skb,
> - const struct sock_filter *fentry)
> +unsigned int __sk_run_filter(void *ctx, const struct sock_filter_int *insn)
> {
> + u64 stack[MAX_BPF_STACK / sizeof(u64)];
> + u64 regs[MAX_BPF_REG], tmp;
> void *ptr;
> - u32 A = 0; /* Accumulator */
> - u32 X = 0; /* Index Register */
> - u32 mem[BPF_MEMWORDS]; /* Scratch Memory Store */
> - u32 tmp;
> - int k;
> + int off;
> +
> +#define K insn->imm
> +#define A regs[insn->a_reg]
> +#define X regs[insn->x_reg]
> +
> +#define CONT ({insn++; goto select_insn; })
> +#define CONT_JMP ({insn++; goto select_insn; })
> +
> + static const void *jumptable[256] = {
> + [0 ... 255] = &&default_label,
> + /* Overwrite non-defaults ... */
> +#define DL(A, B, C) [A|B|C] = &&A##_##B##_##C
> + DL(BPF_ALU, BPF_ADD, BPF_X),
> + DL(BPF_ALU, BPF_ADD, BPF_K),
> + DL(BPF_ALU, BPF_SUB, BPF_X),
> + DL(BPF_ALU, BPF_SUB, BPF_K),
> + DL(BPF_ALU, BPF_AND, BPF_X),
> + DL(BPF_ALU, BPF_AND, BPF_K),
> + DL(BPF_ALU, BPF_OR, BPF_X),
> + DL(BPF_ALU, BPF_OR, BPF_K),
> + DL(BPF_ALU, BPF_LSH, BPF_X),
> + DL(BPF_ALU, BPF_LSH, BPF_K),
> + DL(BPF_ALU, BPF_RSH, BPF_X),
> + DL(BPF_ALU, BPF_RSH, BPF_K),
> + DL(BPF_ALU, BPF_XOR, BPF_X),
> + DL(BPF_ALU, BPF_XOR, BPF_K),
> + DL(BPF_ALU, BPF_MUL, BPF_X),
> + DL(BPF_ALU, BPF_MUL, BPF_K),
> + DL(BPF_ALU, BPF_MOV, BPF_X),
> + DL(BPF_ALU, BPF_MOV, BPF_K),
> + DL(BPF_ALU, BPF_DIV, BPF_X),
> + DL(BPF_ALU, BPF_DIV, BPF_K),
> + DL(BPF_ALU, BPF_MOD, BPF_X),
> + DL(BPF_ALU, BPF_MOD, BPF_K),
> + DL(BPF_ALU, BPF_BSWAP, BPF_X),
> + DL(BPF_ALU, BPF_NEG, 0),
> + DL(BPF_ALU64, BPF_ADD, BPF_X),
> + DL(BPF_ALU64, BPF_ADD, BPF_K),
> + DL(BPF_ALU64, BPF_SUB, BPF_X),
> + DL(BPF_ALU64, BPF_SUB, BPF_K),
> + DL(BPF_ALU64, BPF_AND, BPF_X),
> + DL(BPF_ALU64, BPF_AND, BPF_K),
> + DL(BPF_ALU64, BPF_OR, BPF_X),
> + DL(BPF_ALU64, BPF_OR, BPF_K),
> + DL(BPF_ALU64, BPF_LSH, BPF_X),
> + DL(BPF_ALU64, BPF_LSH, BPF_K),
> + DL(BPF_ALU64, BPF_RSH, BPF_X),
> + DL(BPF_ALU64, BPF_RSH, BPF_K),
> + DL(BPF_ALU64, BPF_XOR, BPF_X),
> + DL(BPF_ALU64, BPF_XOR, BPF_K),
> + DL(BPF_ALU64, BPF_MUL, BPF_X),
> + DL(BPF_ALU64, BPF_MUL, BPF_K),
> + DL(BPF_ALU64, BPF_MOV, BPF_X),
> + DL(BPF_ALU64, BPF_MOV, BPF_K),
> + DL(BPF_ALU64, BPF_ARSH, BPF_X),
> + DL(BPF_ALU64, BPF_ARSH, BPF_K),
> + DL(BPF_ALU64, BPF_DIV, BPF_X),
> + DL(BPF_ALU64, BPF_DIV, BPF_K),
> + DL(BPF_ALU64, BPF_MOD, BPF_X),
> + DL(BPF_ALU64, BPF_MOD, BPF_K),
> + DL(BPF_ALU64, BPF_BSWAP, BPF_X),
> + DL(BPF_ALU64, BPF_NEG, 0),
> + DL(BPF_JMP, BPF_CALL, 0),
> + DL(BPF_JMP, BPF_JA, 0),
> + DL(BPF_JMP, BPF_JEQ, BPF_X),
> + DL(BPF_JMP, BPF_JEQ, BPF_K),
> + DL(BPF_JMP, BPF_JNE, BPF_X),
> + DL(BPF_JMP, BPF_JNE, BPF_K),
> + DL(BPF_JMP, BPF_JGT, BPF_X),
> + DL(BPF_JMP, BPF_JGT, BPF_K),
> + DL(BPF_JMP, BPF_JGE, BPF_X),
> + DL(BPF_JMP, BPF_JGE, BPF_K),
> + DL(BPF_JMP, BPF_JSGT, BPF_X),
> + DL(BPF_JMP, BPF_JSGT, BPF_K),
> + DL(BPF_JMP, BPF_JSGE, BPF_X),
> + DL(BPF_JMP, BPF_JSGE, BPF_K),
> + DL(BPF_JMP, BPF_JSET, BPF_X),
> + DL(BPF_JMP, BPF_JSET, BPF_K),
> + DL(BPF_STX, BPF_MEM, BPF_B),
> + DL(BPF_STX, BPF_MEM, BPF_H),
> + DL(BPF_STX, BPF_MEM, BPF_W),
> + DL(BPF_STX, BPF_MEM, BPF_DW),
> + DL(BPF_ST, BPF_MEM, BPF_B),
> + DL(BPF_ST, BPF_MEM, BPF_H),
> + DL(BPF_ST, BPF_MEM, BPF_W),
> + DL(BPF_ST, BPF_MEM, BPF_DW),
> + DL(BPF_LDX, BPF_MEM, BPF_B),
> + DL(BPF_LDX, BPF_MEM, BPF_H),
> + DL(BPF_LDX, BPF_MEM, BPF_W),
> + DL(BPF_LDX, BPF_MEM, BPF_DW),
> + DL(BPF_STX, BPF_XADD, BPF_W),
> + DL(BPF_STX, BPF_XADD, BPF_DW),
> + DL(BPF_LD, BPF_ABS, BPF_W),
> + DL(BPF_LD, BPF_ABS, BPF_H),
> + DL(BPF_LD, BPF_ABS, BPF_B),
> + DL(BPF_LD, BPF_IND, BPF_W),
> + DL(BPF_LD, BPF_IND, BPF_H),
> + DL(BPF_LD, BPF_IND, BPF_B),
> + DL(BPF_RET, BPF_K, 0),
> +#undef DL
> + };
>
> - /*
> - * Process array of filter instructions.
> - */
> - for (;; fentry++) {
> -#if defined(CONFIG_X86_32)
> -#define K (fentry->k)
> -#else
> - const u32 K = fentry->k;
> -#endif
> -
> - switch (fentry->code) {
> - case BPF_S_ALU_ADD_X:
> - A += X;
> - continue;
> - case BPF_S_ALU_ADD_K:
> - A += K;
> - continue;
> - case BPF_S_ALU_SUB_X:
> - A -= X;
> - continue;
> - case BPF_S_ALU_SUB_K:
> - A -= K;
> - continue;
> - case BPF_S_ALU_MUL_X:
> - A *= X;
> - continue;
> - case BPF_S_ALU_MUL_K:
> - A *= K;
> - continue;
> - case BPF_S_ALU_DIV_X:
> - if (X == 0)
> - return 0;
> - A /= X;
> - continue;
> - case BPF_S_ALU_DIV_K:
> - A /= K;
> - continue;
> - case BPF_S_ALU_MOD_X:
> - if (X == 0)
> - return 0;
> - A %= X;
> - continue;
> - case BPF_S_ALU_MOD_K:
> - A %= K;
> - continue;
> - case BPF_S_ALU_AND_X:
> - A &= X;
> - continue;
> - case BPF_S_ALU_AND_K:
> - A &= K;
> - continue;
> - case BPF_S_ALU_OR_X:
> - A |= X;
> - continue;
> - case BPF_S_ALU_OR_K:
> - A |= K;
> - continue;
> - case BPF_S_ANC_ALU_XOR_X:
> - case BPF_S_ALU_XOR_X:
> - A ^= X;
> - continue;
> - case BPF_S_ALU_XOR_K:
> - A ^= K;
> - continue;
> - case BPF_S_ALU_LSH_X:
> - A <<= X;
> - continue;
> - case BPF_S_ALU_LSH_K:
> - A <<= K;
> - continue;
> - case BPF_S_ALU_RSH_X:
> - A >>= X;
> - continue;
> - case BPF_S_ALU_RSH_K:
> - A >>= K;
> - continue;
> - case BPF_S_ALU_NEG:
> - A = -A;
> - continue;
> - case BPF_S_JMP_JA:
> - fentry += K;
> - continue;
> - case BPF_S_JMP_JGT_K:
> - fentry += (A > K) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JGE_K:
> - fentry += (A >= K) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JEQ_K:
> - fentry += (A == K) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JSET_K:
> - fentry += (A & K) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JGT_X:
> - fentry += (A > X) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JGE_X:
> - fentry += (A >= X) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JEQ_X:
> - fentry += (A == X) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_JMP_JSET_X:
> - fentry += (A & X) ? fentry->jt : fentry->jf;
> - continue;
> - case BPF_S_LD_W_ABS:
> - k = K;
> -load_w:
> - ptr = load_pointer(skb, k, 4, &tmp);
> - if (ptr != NULL) {
> - A = get_unaligned_be32(ptr);
> - continue;
> - }
> - return 0;
> - case BPF_S_LD_H_ABS:
> - k = K;
> -load_h:
> - ptr = load_pointer(skb, k, 2, &tmp);
> - if (ptr != NULL) {
> - A = get_unaligned_be16(ptr);
> - continue;
> + regs[FP_REG] = (u64) (unsigned long) &stack[ARRAY_SIZE(stack)];
> + regs[CTX_REG] = (u64) (unsigned long) ctx;
> +
> +select_insn:
> + goto *jumptable[insn->code];
> +
> + /* ALU */
> +#define ALU(OPCODE, OP) \
> + BPF_ALU64_##OPCODE##_BPF_X: \
> + A = A OP X; \
> + CONT; \
> + BPF_ALU_##OPCODE##_BPF_X: \
> + A = (u32) A OP (u32) X; \
> + CONT; \
> + BPF_ALU64_##OPCODE##_BPF_K: \
> + A = A OP K; \
> + CONT; \
> + BPF_ALU_##OPCODE##_BPF_K: \
> + A = (u32) A OP (u32) K; \
> + CONT;
> +
> + ALU(BPF_ADD, +)
> + ALU(BPF_SUB, -)
> + ALU(BPF_AND, &)
> + ALU(BPF_OR, |)
> + ALU(BPF_LSH, <<)
> + ALU(BPF_RSH, >>)
> + ALU(BPF_XOR, ^)
> + ALU(BPF_MUL, *)
> +#undef ALU
> + BPF_ALU_BPF_NEG_0:
> + A = (u32) -A;
> + CONT;
> + BPF_ALU64_BPF_NEG_0:
> + A = -A;
> + CONT;
> + BPF_ALU_BPF_MOV_BPF_X:
> + A = (u32) X;
> + CONT;
> + BPF_ALU_BPF_MOV_BPF_K:
> + A = (u32) K;
> + CONT;
> + BPF_ALU64_BPF_MOV_BPF_X:
> + A = X;
> + CONT;
> + BPF_ALU64_BPF_MOV_BPF_K:
> + A = K;
> + CONT;
> + BPF_ALU64_BPF_ARSH_BPF_X:
> + (*(s64 *) &A) >>= X;
> + CONT;
> + BPF_ALU64_BPF_ARSH_BPF_K:
> + (*(s64 *) &A) >>= K;
> + CONT;
> + BPF_ALU64_BPF_MOD_BPF_X:
> + tmp = A;
> + if (X)
> + A = do_div(tmp, X);
> + CONT;
> + BPF_ALU_BPF_MOD_BPF_X:
> + tmp = (u32) A;
> + if (X)
> + A = do_div(tmp, (u32) X);
> + CONT;
> + BPF_ALU64_BPF_MOD_BPF_K:
> + tmp = A;
> + if (K)
> + A = do_div(tmp, K);
> + CONT;
> + BPF_ALU_BPF_MOD_BPF_K:
> + tmp = (u32) A;
> + if (K)
> + A = do_div(tmp, (u32) K);
> + CONT;
> + BPF_ALU64_BPF_DIV_BPF_X:
> + if (X)
> + do_div(A, X);
> + CONT;
> + BPF_ALU_BPF_DIV_BPF_X:
> + tmp = (u32) A;
> + if (X)
> + do_div(tmp, (u32) X);
> + A = (u32) tmp;
> + CONT;
> + BPF_ALU64_BPF_DIV_BPF_K:
> + if (K)
> + do_div(A, K);
> + CONT;
> + BPF_ALU_BPF_DIV_BPF_K:
> + tmp = (u32) A;
> + if (K)
> + do_div(tmp, (u32) K);
> + A = (u32) tmp;
> + CONT;
> + BPF_ALU_BPF_BSWAP_BPF_X:
> + A = swab32(A);
> + CONT;
> + BPF_ALU64_BPF_BSWAP_BPF_X:
> + A = swab64(A);
> + CONT;
> +
> + /* CALL */
> + BPF_JMP_BPF_CALL_0:
> + regs[0] = (__bpf_call_base + insn->imm)(regs[1], regs[2],
> + regs[3], regs[4],
> + regs[5]);
> + CONT;
> +
> + /* JMP */
> + BPF_JMP_BPF_JA_0:
> + insn += insn->off;
> + CONT;
> + BPF_JMP_BPF_JEQ_BPF_X:
> + if (A == X) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JEQ_BPF_K:
> + if (A == K) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JNE_BPF_X:
> + if (A != X) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JNE_BPF_K:
> + if (A != K) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JGT_BPF_X:
> + if (A > X) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JGT_BPF_K:
> + if (A > K) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JGE_BPF_X:
> + if (A >= X) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JGE_BPF_K:
> + if (A >= K) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSGT_BPF_X:
> + if (((s64)A) > ((s64)X)) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSGT_BPF_K:
> + if (((s64)A) > ((s64)K)) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSGE_BPF_X:
> + if (((s64)A) >= ((s64)X)) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSGE_BPF_K:
> + if (((s64)A) >= ((s64)K)) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSET_BPF_X:
> + if (A & X) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> + BPF_JMP_BPF_JSET_BPF_K:
> + if (A & K) {
> + insn += insn->off;
> + CONT_JMP;
> + }
> + CONT;
> +
> + /* STX and ST and LDX*/
> +#define LDST(SIZEOP, SIZE) \
> + BPF_STX_BPF_MEM_##SIZEOP: \
> + *(SIZE *)(unsigned long) (A + insn->off) = X; \
> + CONT; \
> + BPF_ST_BPF_MEM_##SIZEOP: \
> + *(SIZE *)(unsigned long) (A + insn->off) = K; \
> + CONT; \
> + BPF_LDX_BPF_MEM_##SIZEOP: \
> + A = *(SIZE *)(unsigned long) (X + insn->off); \
> + CONT;
> +
> + LDST(BPF_B, u8)
> + LDST(BPF_H, u16)
> + LDST(BPF_W, u32)
> + LDST(BPF_DW, u64)
> +#undef LDST
> + BPF_STX_BPF_XADD_BPF_W: /* lock xadd *(u32 *)(A + insn->off) += X */
> + atomic_add((u32) X, (atomic_t *)(unsigned long)
> + (A + insn->off));
> + CONT;
> + BPF_STX_BPF_XADD_BPF_DW: /* lock xadd *(u64 *)(A + insn->off) += X */
> + atomic64_add((u64) X, (atomic64_t *)(unsigned long)
> + (A + insn->off));
> + CONT;
> + BPF_LD_BPF_ABS_BPF_W: /* A = *(u32 *)(ctx + K) */
> + off = K;
> +load_word:
> + /* BPF_LD + BPD_ABS and BPF_LD + BPF_IND insns are only
> + * appearing in the programs where ctx == skb.
> + */
> + ptr = load_pointer((struct sk_buff *) ctx, off, 4, &tmp);
> + if (likely(ptr != NULL)) {
> + A = get_unaligned_be32(ptr);
> + CONT;
> + }
> + return 0;
> + BPF_LD_BPF_ABS_BPF_H: /* A = *(u16 *)(ctx + K) */
> + off = K;
> +load_half:
> + ptr = load_pointer((struct sk_buff *) ctx, off, 2, &tmp);
> + if (likely(ptr != NULL)) {
> + A = get_unaligned_be16(ptr);
> + CONT;
> + }
> + return 0;
> +
> + BPF_LD_BPF_ABS_BPF_B: /* A = *(u8 *)(ctx + K) */
> + off = K;
> +load_byte:
> + ptr = load_pointer((struct sk_buff *) ctx, off, 1, &tmp);
> + if (likely(ptr != NULL)) {
> + A = *(u8 *)ptr;
> + CONT;
> + }
> + return 0;
> + BPF_LD_BPF_IND_BPF_W: /* A = *(u32 *)(ctx + X + K) */
> + off = K + X;
> + goto load_word;
> + BPF_LD_BPF_IND_BPF_H: /* A = *(u16 *)(ctx + X + K) */
> + off = K + X;
> + goto load_half;
> + BPF_LD_BPF_IND_BPF_B: /* A = *(u8 *)(ctx + X + K) */
> + off = K + X;
> + goto load_byte;
> +
> + /* RET */
> + BPF_RET_BPF_K_0:
> + return regs[0 /* R0 */];
> +
> + default_label:
> + /* If we ever reach this, we have a bug somewhere. */
> + WARN_RATELIMIT(1, "unknown opcode %02x\n", insn->code);
> + return 0;
> +#undef CONT_JMP
> +#undef CONT
> +#undef A
> +#undef X
> +#undef K
> +}
> +
> +u32 sk_run_filter_int_seccomp(const struct seccomp_data *ctx,
> + const struct sock_filter_int *insni)
> + __attribute__ ((alias ("__sk_run_filter")));
> +
> +u32 sk_run_filter_int_skb(const struct sk_buff *ctx,
> + const struct sock_filter_int *insni)
> + __attribute__ ((alias ("__sk_run_filter")));
> +EXPORT_SYMBOL_GPL(sk_run_filter_int_skb);
> +
> +/* Helper to find the offset of pkt_type in sk_buff structure. We want
> + * to make sure its still a 3bit field starting at a byte boundary;
> + * taken from arch/x86/net/bpf_jit_comp.c.
> + */
> +#define PKT_TYPE_MAX 7
> +static unsigned int pkt_type_offset(void)
> +{
> + struct sk_buff skb_probe = { .pkt_type = ~0, };
> + u8 *ct = (u8 *) &skb_probe;
> + unsigned int off;
> +
> + for (off = 0; off < sizeof(struct sk_buff); off++) {
> + if (ct[off] == PKT_TYPE_MAX)
> + return off;
> + }
> +
> + pr_err_once("Please fix %s, as pkt_type couldn't be found!\n", __func__);
> + return -1;
> +}
> +
> +static u64 __skb_get_pay_offset(u64 ctx, u64 A, u64 X, u64 r4, u64 r5)
> +{
> + struct sk_buff *skb = (struct sk_buff *)(long) ctx;
> +
> + return __skb_get_poff(skb);
> +}
> +
> +static u64 __skb_get_nlattr(u64 ctx, u64 A, u64 X, u64 r4, u64 r5)
> +{
> + struct sk_buff *skb = (struct sk_buff *)(long) ctx;
> + struct nlattr *nla;
> +
> + if (skb_is_nonlinear(skb))
> + return 0;
> +
> + if (A > skb->len - sizeof(struct nlattr))
> + return 0;
> +
> + nla = nla_find((struct nlattr *) &skb->data[A], skb->len - A, X);
> + if (nla)
> + return (void *) nla - (void *) skb->data;
> +
> + return 0;
> +}
> +
> +static u64 __skb_get_nlattr_nest(u64 ctx, u64 A, u64 X, u64 r4, u64 r5)
> +{
> + struct sk_buff *skb = (struct sk_buff *)(long) ctx;
> + struct nlattr *nla;
> +
> + if (skb_is_nonlinear(skb))
> + return 0;
> +
> + if (A > skb->len - sizeof(struct nlattr))
> + return 0;
> +
> + nla = (struct nlattr *) &skb->data[A];
> + if (nla->nla_len > A - skb->len)
> + return 0;
> +
> + nla = nla_find_nested(nla, X);
> + if (nla)
> + return (void *) nla - (void *) skb->data;
> +
> + return 0;
> +}
> +
> +static u64 __get_raw_cpu_id(u64 ctx, u64 A, u64 X, u64 r4, u64 r5)
> +{
> + return raw_smp_processor_id();
> +}
> +
> +/* Register mappings for user programs. */
> +#define A_REG 6
> +#define X_REG 7
> +#define TMP_REG 8
> +
> +static bool convert_bpf_extensions(struct sock_filter *fp,
> + struct sock_filter_int **insnp)
> +{
> + struct sock_filter_int *insn = *insnp;
> +
> + switch (fp->k) {
> + case SKF_AD_OFF + SKF_AD_PROTOCOL:
> + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, protocol) != 2);
> +
> + insn->code = BPF_LDX | BPF_MEM | BPF_H;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, protocol);
> +#ifdef __LITTLE_ENDIAN
> + insn++;
> +
> + /* A = swab32(A) */
> + insn->code = BPF_ALU | BPF_BSWAP | BPF_X;
> + insn->a_reg = A_REG;
> + insn++;
> +
> + /* A >>= 16 */
> + insn->code = BPF_ALU | BPF_RSH | BPF_K;
> + insn->a_reg = A_REG;
> + insn->imm = 16;
> +#endif /* __LITTLE_ENDIAN */
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_PKTTYPE:
> + insn->code = BPF_LDX | BPF_MEM | BPF_B;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = pkt_type_offset();
> + if (insn->off < 0)
> + return false;
> + insn++;
> +
> + insn->code = BPF_ALU | BPF_AND | BPF_K;
> + insn->a_reg = A_REG;
> + insn->imm = PKT_TYPE_MAX;
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_IFINDEX:
> + case SKF_AD_OFF + SKF_AD_HATYPE:
> + if (FIELD_SIZEOF(struct sk_buff, dev) == 8)
> + insn->code = BPF_LDX | BPF_MEM | BPF_DW;
> + else
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = TMP_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, dev);
> + insn++;
> +
> + insn->code = BPF_JMP | BPF_JNE | BPF_K;
> + insn->a_reg = TMP_REG;
> + insn->imm = 0;
> + insn->off = 1;
> + insn++;
> +
> + insn->code = BPF_RET | BPF_K;
> + insn++;
> +
> + BUILD_BUG_ON(FIELD_SIZEOF(struct net_device, ifindex) != 4);
> + BUILD_BUG_ON(FIELD_SIZEOF(struct net_device, type) != 2);
> +
> + insn->a_reg = A_REG;
> + insn->x_reg = TMP_REG;
> +
> + if (fp->k == SKF_AD_OFF + SKF_AD_IFINDEX) {
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->off = offsetof(struct net_device, ifindex);
> + } else {
> + insn->code = BPF_LDX | BPF_MEM | BPF_H;
> + insn->off = offsetof(struct net_device, type);
> + }
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_MARK:
> + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, mark) != 4);
> +
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, mark);
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_RXHASH:
> + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, rxhash) != 4);
> +
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, rxhash);
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_QUEUE:
> + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, queue_mapping) != 2);
> +
> + insn->code = BPF_LDX | BPF_MEM | BPF_H;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, queue_mapping);
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_VLAN_TAG:
> + case SKF_AD_OFF + SKF_AD_VLAN_TAG_PRESENT:
> + BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, vlan_tci) != 2);
> +
> + insn->code = BPF_LDX | BPF_MEM | BPF_H;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, vlan_tci);
> + insn++;
> +
> + BUILD_BUG_ON(VLAN_TAG_PRESENT != 0x1000);
> +
> + if (fp->k == SKF_AD_OFF + SKF_AD_VLAN_TAG) {
> + insn->code = BPF_ALU | BPF_AND | BPF_K;
> + insn->a_reg = A_REG;
> + insn->imm = ~VLAN_TAG_PRESENT;
> + } else {
> + insn->code = BPF_ALU | BPF_RSH | BPF_K;
> + insn->a_reg = A_REG;
> + insn->imm = 12;
> + insn++;
> +
> + insn->code = BPF_ALU | BPF_AND | BPF_K;
> + insn->a_reg = A_REG;
> + insn->imm = 1;
> + }
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_PAY_OFFSET:
> + case SKF_AD_OFF + SKF_AD_NLATTR:
> + case SKF_AD_OFF + SKF_AD_NLATTR_NEST:
> + case SKF_AD_OFF + SKF_AD_CPU:
> + /* Save ctx */
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = TMP_REG;
> + insn->x_reg = CTX_REG;
> + insn++;
> +
> + /* arg2 = A */
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = 2;
> + insn->x_reg = A_REG;
> + insn++;
> +
> + /* arg3 = X */
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = 3;
> + insn->x_reg = X_REG;
> + insn++;
> +
> + /* Emit call(ctx, arg2=A, arg3=X) */
> + insn->code = BPF_JMP | BPF_CALL;
> + /* Re: sparse ... Share your drugs? High on caffeine ... ;-) */
> + switch (fp->k) {
> + case SKF_AD_OFF + SKF_AD_PAY_OFFSET:
> + insn->imm = __skb_get_pay_offset - __bpf_call_base;
> + break;
> + case SKF_AD_OFF + SKF_AD_NLATTR:
> + insn->imm = __skb_get_nlattr - __bpf_call_base;
> + break;
> + case SKF_AD_OFF + SKF_AD_NLATTR_NEST:
> + insn->imm = __skb_get_nlattr_nest - __bpf_call_base;
> + break;
> + case SKF_AD_OFF + SKF_AD_CPU:
> + insn->imm = __get_raw_cpu_id - __bpf_call_base;
> + break;
> + }
> + insn++;
> +
> + /* Restore ctx */
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = CTX_REG;
> + insn->x_reg = TMP_REG;
> + insn++;
> +
> + /* Move ret value into A_REG */
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = A_REG;
> + insn->x_reg = 0;
> + break;
> +
> + case SKF_AD_OFF + SKF_AD_ALU_XOR_X:
> + insn->code = BPF_ALU | BPF_XOR | BPF_X;
> + insn->a_reg = A_REG;
> + insn->x_reg = X_REG;
> + break;
> +
> + default:
> + /* This is just a dummy call to avoid letting the compiler
> + * evict __bpf_call_base() as an optimization. Placed here
> + * where no-one bothers.
> + */
> + BUG_ON(__bpf_call_base(0, 0, 0, 0, 0) != 0);
> + return false;
> + }
> +
> + *insnp = insn;
> + return true;
> +}
> +
> +/**
> + * sk_convert_filter - convert filter program
> + * @prog: the user passed filter program
> + * @len: the length of the user passed filter program
> + * @new_prog: buffer where converted program will be stored
> + * @new_len: pointer to store length of converted program
> + *
> + * Remap 'sock_filter' style BPF instruction set to 'sock_filter_ext' style.
> + * Conversion workflow:
> + *
> + * 1) First pass for calculating the new program length:
> + * sk_convert_filter(old_prog, old_len, NULL, &new_len)
> + *
> + * 2) 2nd pass to remap in two passes: 1st pass finds new
> + * jump offsets, 2nd pass remapping:
> + * new_prog = kmalloc(sizeof(struct sock_filter_int) * new_len);
> + * sk_convert_filter(old_prog, old_len, new_prog, &new_len);
> + *
> + * User BPF's register A is mapped to our BPF register 6, user BPF
> + * register X is mapped to BPF register 7; frame pointer is always
> + * register 10; Context 'void *ctx' is stored in register 1, that is,
> + * for socket filters: ctx == 'struct sk_buff *', for seccomp:
> + * ctx == 'struct seccomp_data *'.
> + */
> +int sk_convert_filter(struct sock_filter *prog, int len,
> + struct sock_filter_int *new_prog, int *new_len)
> +{
> + int new_flen = 0, pass = 0, target, i;
> + struct sock_filter_int *new_insn;
> + struct sock_filter *fp;
> + int *addrs = NULL;
> + u8 bpf_src;
> +
> + BUILD_BUG_ON(BPF_MEMWORDS * sizeof(u32) > MAX_BPF_STACK);
> + BUILD_BUG_ON(FP_REG + 1 != MAX_BPF_REG);
> +
> + if (len <= 0 || len >= BPF_MAXINSNS)
> + return -EINVAL;
> +
> + if (new_prog) {
> + addrs = kzalloc(len * sizeof(*addrs), GFP_KERNEL);
> + if (!addrs)
> + return -ENOMEM;
> + }
> +
> +do_pass:
> + new_insn = new_prog;
> + fp = prog;
> +
> + for (i = 0; i < len; fp++, i++) {
> + struct sock_filter_int tmp_insns[6] = { };
> + struct sock_filter_int *insn = tmp_insns;
> +
> + if (addrs)
> + addrs[i] = new_insn - new_prog;
> +
> + switch (fp->code) {
> + /* All arithmetic insns and skb loads map as-is. */
> + case BPF_ALU | BPF_ADD | BPF_X:
> + case BPF_ALU | BPF_ADD | BPF_K:
> + case BPF_ALU | BPF_SUB | BPF_X:
> + case BPF_ALU | BPF_SUB | BPF_K:
> + case BPF_ALU | BPF_AND | BPF_X:
> + case BPF_ALU | BPF_AND | BPF_K:
> + case BPF_ALU | BPF_OR | BPF_X:
> + case BPF_ALU | BPF_OR | BPF_K:
> + case BPF_ALU | BPF_LSH | BPF_X:
> + case BPF_ALU | BPF_LSH | BPF_K:
> + case BPF_ALU | BPF_RSH | BPF_X:
> + case BPF_ALU | BPF_RSH | BPF_K:
> + case BPF_ALU | BPF_XOR | BPF_X:
> + case BPF_ALU | BPF_XOR | BPF_K:
> + case BPF_ALU | BPF_MUL | BPF_X:
> + case BPF_ALU | BPF_MUL | BPF_K:
> + case BPF_ALU | BPF_DIV | BPF_X:
> + case BPF_ALU | BPF_DIV | BPF_K:
> + case BPF_ALU | BPF_MOD | BPF_X:
> + case BPF_ALU | BPF_MOD | BPF_K:
> + case BPF_ALU | BPF_NEG:
> + case BPF_LD | BPF_ABS | BPF_W:
> + case BPF_LD | BPF_ABS | BPF_H:
> + case BPF_LD | BPF_ABS | BPF_B:
> + case BPF_LD | BPF_IND | BPF_W:
> + case BPF_LD | BPF_IND | BPF_H:
> + case BPF_LD | BPF_IND | BPF_B:
> + /* Check for overloaded BPF extension and
> + * directly convert it if found, otherwise
> + * just move on with mapping.
> + */
> + if (BPF_CLASS(fp->code) == BPF_LD &&
> + BPF_MODE(fp->code) == BPF_ABS &&
> + convert_bpf_extensions(fp, &insn))
> + break;
> +
> + insn->code = fp->code;
> + insn->a_reg = A_REG;
> + insn->x_reg = X_REG;
> + insn->imm = fp->k;
> + break;
> +
> + /* Jump opcodes map as-is, but offsets need adjustment. */
> + case BPF_JMP | BPF_JA:
> + target = i + fp->k + 1;
> + insn->code = fp->code;
> +#define EMIT_JMP \
> + do { \
> + if (target >= len || target < 0) \
> + goto err; \
> + insn->off = addrs ? addrs[target] - addrs[i] - 1 : 0; \
> + /* Adjust pc relative offset for 2nd or 3rd insn. */ \
> + insn->off -= insn - tmp_insns; \
> + } while (0)
> +
> + EMIT_JMP;
> + break;
> +
> + case BPF_JMP | BPF_JEQ | BPF_K:
> + case BPF_JMP | BPF_JEQ | BPF_X:
> + case BPF_JMP | BPF_JSET | BPF_K:
> + case BPF_JMP | BPF_JSET | BPF_X:
> + case BPF_JMP | BPF_JGT | BPF_K:
> + case BPF_JMP | BPF_JGT | BPF_X:
> + case BPF_JMP | BPF_JGE | BPF_K:
> + case BPF_JMP | BPF_JGE | BPF_X:
> + if (BPF_SRC(fp->code) == BPF_K && (int) fp->k < 0) {
> + /* BPF immediates are signed, zero extend
> + * immediate into tmp register and use it
> + * in compare insn.
> + */
> + insn->code = BPF_ALU | BPF_MOV | BPF_K;
> + insn->a_reg = TMP_REG;
> + insn->imm = fp->k;
> + insn++;
> +
> + insn->a_reg = A_REG;
> + insn->x_reg = TMP_REG;
> + bpf_src = BPF_X;
> + } else {
> + insn->a_reg = A_REG;
> + insn->x_reg = X_REG;
> + insn->imm = fp->k;
> + bpf_src = BPF_SRC(fp->code);
> }
> - return 0;
> - case BPF_S_LD_B_ABS:
> - k = K;
> -load_b:
> - ptr = load_pointer(skb, k, 1, &tmp);
> - if (ptr != NULL) {
> - A = *(u8 *)ptr;
> - continue;
> +
> + /* Common case where 'jump_false' is next insn. */
> + if (fp->jf == 0) {
> + insn->code = BPF_JMP | BPF_OP(fp->code) | bpf_src;
> + target = i + fp->jt + 1;
> + EMIT_JMP;
> + break;
> }
> - return 0;
> - case BPF_S_LD_W_LEN:
> - A = skb->len;
> - continue;
> - case BPF_S_LDX_W_LEN:
> - X = skb->len;
> - continue;
> - case BPF_S_LD_W_IND:
> - k = X + K;
> - goto load_w;
> - case BPF_S_LD_H_IND:
> - k = X + K;
> - goto load_h;
> - case BPF_S_LD_B_IND:
> - k = X + K;
> - goto load_b;
> - case BPF_S_LDX_B_MSH:
> - ptr = load_pointer(skb, K, 1, &tmp);
> - if (ptr != NULL) {
> - X = (*(u8 *)ptr & 0xf) << 2;
> - continue;
> +
> + /* Convert JEQ into JNE when 'jump_true' is next insn. */
> + if (fp->jt == 0 && BPF_OP(fp->code) == BPF_JEQ) {
> + insn->code = BPF_JMP | BPF_JNE | bpf_src;
> + target = i + fp->jf + 1;
> + EMIT_JMP;
> + break;
> }
> - return 0;
> - case BPF_S_LD_IMM:
> - A = K;
> - continue;
> - case BPF_S_LDX_IMM:
> - X = K;
> - continue;
> - case BPF_S_LD_MEM:
> - A = mem[K];
> - continue;
> - case BPF_S_LDX_MEM:
> - X = mem[K];
> - continue;
> - case BPF_S_MISC_TAX:
> - X = A;
> - continue;
> - case BPF_S_MISC_TXA:
> - A = X;
> - continue;
> - case BPF_S_RET_K:
> - return K;
> - case BPF_S_RET_A:
> - return A;
> - case BPF_S_ST:
> - mem[K] = A;
> - continue;
> - case BPF_S_STX:
> - mem[K] = X;
> - continue;
> - case BPF_S_ANC_PROTOCOL:
> - A = ntohs(skb->protocol);
> - continue;
> - case BPF_S_ANC_PKTTYPE:
> - A = skb->pkt_type;
> - continue;
> - case BPF_S_ANC_IFINDEX:
> - if (!skb->dev)
> - return 0;
> - A = skb->dev->ifindex;
> - continue;
> - case BPF_S_ANC_MARK:
> - A = skb->mark;
> - continue;
> - case BPF_S_ANC_QUEUE:
> - A = skb->queue_mapping;
> - continue;
> - case BPF_S_ANC_HATYPE:
> - if (!skb->dev)
> - return 0;
> - A = skb->dev->type;
> - continue;
> - case BPF_S_ANC_RXHASH:
> - A = skb->rxhash;
> - continue;
> - case BPF_S_ANC_CPU:
> - A = raw_smp_processor_id();
> - continue;
> - case BPF_S_ANC_VLAN_TAG:
> - A = vlan_tx_tag_get(skb);
> - continue;
> - case BPF_S_ANC_VLAN_TAG_PRESENT:
> - A = !!vlan_tx_tag_present(skb);
> - continue;
> - case BPF_S_ANC_PAY_OFFSET:
> - A = __skb_get_poff(skb);
> - continue;
> - case BPF_S_ANC_NLATTR: {
> - struct nlattr *nla;
> -
> - if (skb_is_nonlinear(skb))
> - return 0;
> - if (A > skb->len - sizeof(struct nlattr))
> - return 0;
> -
> - nla = nla_find((struct nlattr *)&skb->data[A],
> - skb->len - A, X);
> - if (nla)
> - A = (void *)nla - (void *)skb->data;
> - else
> - A = 0;
> - continue;
> - }
> - case BPF_S_ANC_NLATTR_NEST: {
> - struct nlattr *nla;
> -
> - if (skb_is_nonlinear(skb))
> - return 0;
> - if (A > skb->len - sizeof(struct nlattr))
> - return 0;
> -
> - nla = (struct nlattr *)&skb->data[A];
> - if (nla->nla_len > A - skb->len)
> - return 0;
> -
> - nla = nla_find_nested(nla, X);
> - if (nla)
> - A = (void *)nla - (void *)skb->data;
> - else
> - A = 0;
> - continue;
> - }
> -#ifdef CONFIG_SECCOMP_FILTER
> - case BPF_S_ANC_SECCOMP_LD_W:
> - A = seccomp_bpf_load(fentry->k);
> - continue;
> -#endif
> +
> + /* Other jumps are mapped into two insns: Jxx and JA. */
> + target = i + fp->jt + 1;
> + insn->code = BPF_JMP | BPF_OP(fp->code) | bpf_src;
> + EMIT_JMP;
> + insn++;
> +
> + insn->code = BPF_JMP | BPF_JA;
> + target = i + fp->jf + 1;
> + EMIT_JMP;
> + break;
> +
> + /* ldxb 4 * ([14] & 0xf) is remaped into 3 insns. */
> + case BPF_LDX | BPF_MSH | BPF_B:
> + insn->code = BPF_LD | BPF_ABS | BPF_B;
> + insn->a_reg = X_REG;
> + insn->imm = fp->k;
> + insn++;
> +
> + insn->code = BPF_ALU | BPF_AND | BPF_K;
> + insn->a_reg = X_REG;
> + insn->imm = 0xf;
> + insn++;
> +
> + insn->code = BPF_ALU | BPF_LSH | BPF_K;
> + insn->a_reg = X_REG;
> + insn->imm = 2;
> + break;
> +
> + /* RET_K, RET_A are remaped into 2 insns. */
> + case BPF_RET | BPF_A:
> + case BPF_RET | BPF_K:
> + insn->code = BPF_ALU | BPF_MOV |
> + (BPF_RVAL(fp->code) == BPF_K ?
> + BPF_K : BPF_X);
> + insn->a_reg = 0;
> + insn->x_reg = A_REG;
> + insn->imm = fp->k;
> + insn++;
> +
> + insn->code = BPF_RET | BPF_K;
> + break;
> +
> + /* Store to stack. */
> + case BPF_ST:
> + case BPF_STX:
> + insn->code = BPF_STX | BPF_MEM | BPF_W;
> + insn->a_reg = FP_REG;
> + insn->x_reg = fp->code == BPF_ST ? A_REG : X_REG;
> + insn->off = -(BPF_MEMWORDS - fp->k) * 4;
> + break;
> +
> + /* Load from stack. */
> + case BPF_LD | BPF_MEM:
> + case BPF_LDX | BPF_MEM:
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = BPF_CLASS(fp->code) == BPF_LD ?
> + A_REG : X_REG;
> + insn->x_reg = FP_REG;
> + insn->off = -(BPF_MEMWORDS - fp->k) * 4;
> + break;
> +
> + /* A = K or X = K */
> + case BPF_LD | BPF_IMM:
> + case BPF_LDX | BPF_IMM:
> + insn->code = BPF_ALU | BPF_MOV | BPF_K;
> + insn->a_reg = BPF_CLASS(fp->code) == BPF_LD ?
> + A_REG : X_REG;
> + insn->imm = fp->k;
> + break;
> +
> + /* X = A */
> + case BPF_MISC | BPF_TAX:
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = X_REG;
> + insn->x_reg = A_REG;
> + break;
> +
> + /* A = X */
> + case BPF_MISC | BPF_TXA:
> + insn->code = BPF_ALU64 | BPF_MOV | BPF_X;
> + insn->a_reg = A_REG;
> + insn->x_reg = X_REG;
> + break;
> +
> + /* A = skb->len or X = skb->len */
> + case BPF_LD | BPF_W | BPF_LEN:
> + case BPF_LDX | BPF_W | BPF_LEN:
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = BPF_CLASS(fp->code) == BPF_LD ?
> + A_REG : X_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = offsetof(struct sk_buff, len);
> + break;
> +
> + /* access seccomp_data fields */
> + case BPF_LDX | BPF_ABS | BPF_W:
> + insn->code = BPF_LDX | BPF_MEM | BPF_W;
> + insn->a_reg = A_REG;
> + insn->x_reg = CTX_REG;
> + insn->off = fp->k;
> + break;
> +
> default:
> - WARN_RATELIMIT(1, "Unknown code:%u jt:%u tf:%u k:%u\n",
> - fentry->code, fentry->jt,
> - fentry->jf, fentry->k);
> - return 0;
> + goto err;
> }
> +
> + insn++;
> + if (new_prog)
> + memcpy(new_insn, tmp_insns,
> + sizeof(*insn) * (insn - tmp_insns));
> +
> + new_insn += insn - tmp_insns;
> }
>
> + if (!new_prog) {
> + /* Only calculating new length. */
> + *new_len = new_insn - new_prog;
> + return 0;
> + }
> +
> + pass++;
> + if (new_flen != new_insn - new_prog) {
> + new_flen = new_insn - new_prog;
> + if (pass > 2)
> + goto err;
> +
> + goto do_pass;
> + }
> +
> + kfree(addrs);
> + BUG_ON(*new_len != new_flen);
> return 0;
> +err:
> + kfree(addrs);
> + return -EINVAL;
> }
> -EXPORT_SYMBOL(sk_run_filter);
>
> -/*
> - * Security :
> +/* Security:
> + *
> * A BPF program is able to use 16 cells of memory to store intermediate
> - * values (check u32 mem[BPF_MEMWORDS] in sk_run_filter())
> + * values (check u32 mem[BPF_MEMWORDS] in sk_run_filter()).
> + *
> * As we dont want to clear mem[] array for each packet going through
> * sk_run_filter(), we check that filter loaded by user never try to read
> * a cell if not previously written, and we check all branches to be sure
> @@ -696,19 +1400,130 @@ void sk_filter_charge(struct sock *sk, struct sk_filter *fp)
> atomic_add(sk_filter_size(fp->len), &sk->sk_omem_alloc);
> }
>
> -static int __sk_prepare_filter(struct sk_filter *fp)
> +static struct sk_filter *__sk_migrate_realloc(struct sk_filter *fp,
> + struct sock *sk,
> + unsigned int len)
> +{
> + struct sk_filter *fp_new;
> +
> + if (sk == NULL)
> + return krealloc(fp, len, GFP_KERNEL);
> +
> + fp_new = sock_kmalloc(sk, len, GFP_KERNEL);
> + if (fp_new) {
> + memcpy(fp_new, fp, sizeof(struct sk_filter));
> + /* As we're kepping orig_prog in fp_new along,
> + * we need to make sure we're not evicting it
> + * from the old fp.
> + */
> + fp->orig_prog = NULL;
> + sk_filter_uncharge(sk, fp);
> + }
> +
> + return fp_new;
> +}
> +
> +static struct sk_filter *__sk_migrate_filter(struct sk_filter *fp,
> + struct sock *sk)
> +{
> + struct sock_filter *old_prog;
> + struct sk_filter *old_fp;
> + int i, err, new_len, old_len = fp->len;
> +
> + /* We are free to overwrite insns et al right here as it
> + * won't be used at this point in time anymore internally
> + * after the migration to the internal BPF instruction
> + * representation.
> + */
> + BUILD_BUG_ON(sizeof(struct sock_filter) !=
> + sizeof(struct sock_filter_int));
> +
> + /* For now, we need to unfiddle BPF_S_* identifiers in place.
> + * This can sooner or later on be subject to removal, e.g. when
> + * JITs have been converted.
> + */
> + for (i = 0; i < fp->len; i++)
> + sk_decode_filter(&fp->insns[i], &fp->insns[i]);
> +
> + /* Conversion cannot happen on overlapping memory areas,
> + * so we need to keep the user BPF around until the 2nd
> + * pass. At this time, the user BPF is stored in fp->insns.
> + */
> + old_prog = kmemdup(fp->insns, old_len * sizeof(struct sock_filter),
> + GFP_KERNEL);
> + if (!old_prog) {
> + err = -ENOMEM;
> + goto out_err;
> + }
> +
> + /* 1st pass: calculate the new program length. */
> + err = sk_convert_filter(old_prog, old_len, NULL, &new_len);
> + if (err)
> + goto out_err_free;
> +
> + /* Expand fp for appending the new filter representation. */
> + old_fp = fp;
> + fp = __sk_migrate_realloc(old_fp, sk, sk_filter_size(new_len));
> + if (!fp) {
> + /* The old_fp is still around in case we couldn't
> + * allocate new memory, so uncharge on that one.
> + */
> + fp = old_fp;
> + err = -ENOMEM;
> + goto out_err_free;
> + }
> +
> + fp->bpf_func = sk_run_filter_int_skb;
> + fp->len = new_len;
> +
> + /* 2nd pass: remap sock_filter insns into sock_filter_int insns. */
> + err = sk_convert_filter(old_prog, old_len, fp->insnsi, &new_len);
> + if (err)
> + /* 2nd sk_convert_filter() can fail only if it fails
> + * to allocate memory, remapping must succeed. Note,
> + * that at this time old_fp has already been released
> + * by __sk_migrate_realloc().
> + */
> + goto out_err_free;
> +
> + kfree(old_prog);
> + return fp;
> +
> +out_err_free:
> + kfree(old_prog);
> +out_err:
> + /* Rollback filter setup. */
> + if (sk != NULL)
> + sk_filter_uncharge(sk, fp);
> + else
> + kfree(fp);
> + return ERR_PTR(err);
> +}
> +
> +static struct sk_filter *__sk_prepare_filter(struct sk_filter *fp,
> + struct sock *sk)
> {
> int err;
>
> - fp->bpf_func = sk_run_filter;
> + fp->bpf_func = NULL;
> fp->jited = 0;
>
> err = sk_chk_filter(fp->insns, fp->len);
> if (err)
> - return err;
> + return ERR_PTR(err);
>
> + /* Probe if we can JIT compile the filter and if so, do
> + * the compilation of the filter.
> + */
> bpf_jit_compile(fp);
> - return 0;
> +
> + /* JIT compiler couldn't process this filter, so do the
> + * internal BPF translation for the optimized interpreter.
> + */
> + if (!fp->jited)
> + fp = __sk_migrate_filter(fp, sk);
> +
> + return fp;
> }
>
> /**
> @@ -726,7 +1541,6 @@ int sk_unattached_filter_create(struct sk_filter **pfp,
> {
> unsigned int fsize = sk_filter_proglen(fprog);
> struct sk_filter *fp;
> - int err;
>
> /* Make sure new filter is there and in the right amounts. */
> if (fprog->filter == NULL)
> @@ -746,15 +1560,15 @@ int sk_unattached_filter_create(struct sk_filter **pfp,
> */
> fp->orig_prog = NULL;
>
> - err = __sk_prepare_filter(fp);
> - if (err)
> - goto free_mem;
> + /* __sk_prepare_filter() already takes care of uncharging
> + * memory in case something goes wrong.
> + */
> + fp = __sk_prepare_filter(fp, NULL);
> + if (IS_ERR(fp))
> + return PTR_ERR(fp);
>
> *pfp = fp;
> return 0;
> -free_mem:
> - kfree(fp);
> - return err;
> }
> EXPORT_SYMBOL_GPL(sk_unattached_filter_create);
>
> @@ -806,11 +1620,12 @@ int sk_attach_filter(struct sock_fprog *fprog, struct sock *sk)
> return -ENOMEM;
> }
>
> - err = __sk_prepare_filter(fp);
> - if (err) {
> - sk_filter_uncharge(sk, fp);
> - return err;
> - }
> + /* __sk_prepare_filter() already takes care of uncharging
> + * memory in case something goes wrong.
> + */
> + fp = __sk_prepare_filter(fp, sk);
> + if (IS_ERR(fp))
> + return PTR_ERR(fp);
>
> old_fp = rcu_dereference_protected(sk->sk_filter,
> sock_owned_by_user(sk));
> --
> 1.7.11.7
>
--
Kees Cook
Chrome OS Security
^ permalink raw reply [flat|nested] 2+ messages in thread