BUG: corrupted list in __dentry_kill (2)

BUG: corrupted list in __dentry_kill (2)

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    938f49c8 Add linux-next specific files for 20191211
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com

list_del corruption. prev->next should be ffff88808fd17b10, but was  
ffff88808fd17590
------------[ cut here ]------------
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 393 Comm: kworker/u4:5 Not tainted  
5.5.0-rc1-next-20191211-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Workqueue: netns cleanup_net
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 c9 00 cb fd 0f 0b 48 89 f1 48 c7 c7 c0 10 70 88 4c 89 e6 e8 b5 00  
cb fd 0f 0b 4c 89 f6 48 c7 c7 60 12 70 88 e8 a4 00 cb fd <0f> 0b 4c 89 ea  
4c 89 f6 48 c7 c7 a0 11 70 88 e8 90 00 cb fd 0f 0b
RSP: 0018:ffffc90001617980 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880a1ce8840 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815e8576 RDI: fffff520002c2f22
RBP: ffffc90001617998 R08: 0000000000000054 R09: ffffed1015d26621
R10: ffffed1015d26620 R11: ffff8880ae933107 R12: ffff8880a1ce8940
R13: ffff88808fd17590 R14: ffff88808fd17b10 R15: ffff88808fd17b10
FS:  0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffff600400 CR3: 000000008d040000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
  __list_del_entry include/linux/list.h:131 [inline]
  dentry_unlist fs/dcache.c:522 [inline]
  __dentry_kill+0x1fd/0x600 fs/dcache.c:575
  dentry_kill fs/dcache.c:698 [inline]
  dput+0x62f/0xe10 fs/dcache.c:859
  simple_recursive_removal+0x5bc/0x6d0 fs/libfs.c:302
  debugfs_remove fs/debugfs/inode.c:713 [inline]
  debugfs_remove+0x5e/0x80 fs/debugfs/inode.c:707
  nsim_ipsec_teardown+0x7c/0x8f drivers/net/netdevsim/ipsec.c:298
  nsim_destroy+0x42/0x70 drivers/net/netdevsim/netdev.c:331
  __nsim_dev_port_del+0x150/0x1f0 drivers/net/netdevsim/dev.c:674
  nsim_dev_port_del_all+0x8b/0xe0 drivers/net/netdevsim/dev.c:687
  nsim_dev_reload_destroy+0x58/0xf0 drivers/net/netdevsim/dev.c:856
  nsim_dev_reload_down+0x73/0xe0 drivers/net/netdevsim/dev.c:493
  devlink_reload+0xc8/0x3c0 net/core/devlink.c:2797
  devlink_pernet_pre_exit+0x104/0x1a0 net/core/devlink.c:8260
  ops_pre_exit_list net/core/net_namespace.c:162 [inline]
  cleanup_net+0x49b/0xaf0 net/core/net_namespace.c:585
  process_one_work+0x9af/0x1740 kernel/workqueue.c:2264
  worker_thread+0x98/0xe40 kernel/workqueue.c:2410
  kthread+0x361/0x430 kernel/kthread.c:255
  ret_from_fork+0x24/0x30 arch/x86/entry/entry_64.S:352
Modules linked in:
---[ end trace 700329e407063cc0 ]---
RIP: 0010:__list_del_entry_valid.cold+0xf/0x4f lib/list_debug.c:51
Code: e8 c9 00 cb fd 0f 0b 48 89 f1 48 c7 c7 c0 10 70 88 4c 89 e6 e8 b5 00  
cb fd 0f 0b 4c 89 f6 48 c7 c7 60 12 70 88 e8 a4 00 cb fd <0f> 0b 4c 89 ea  
4c 89 f6 48 c7 c7 a0 11 70 88 e8 90 00 cb fd 0f 0b
RSP: 0018:ffffc90001617980 EFLAGS: 00010286
RAX: 0000000000000054 RBX: ffff8880a1ce8840 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffffffff815e8576 RDI: fffff520002c2f22
RBP: ffffc90001617998 R08: 0000000000000054 R09: ffffed1015d26621
R10: ffffed1015d26620 R11: ffff8880ae933107 R12: ffff8880a1ce8940
R13: ffff88808fd17590 R14: ffff88808fd17b10 R15: ffff88808fd17b10
FS:  0000000000000000(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffff600400 CR3: 000000008d040000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
syzbot can test patches for this bug, for details see:
https://goo.gl/tpsmEJ#testing-patches

Re: BUG: corrupted list in __dentry_kill (2)

[Al Viro]

On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> Hello,
> 
> syzbot found the following crash on:
> 
> HEAD commit:    938f49c8 Add linux-next specific files for 20191211
> git tree:       linux-next
> console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> 
> IMPORTANT: if you fix the bug, please add the following tag to the commit:
> Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com

Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672

Re: BUG: corrupted list in __dentry_kill (2)

[Dmitry Vyukov]

On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > Hello,
> >
> > syzbot found the following crash on:
> >
> > HEAD commit:    938f49c8 Add linux-next specific files for 20191211
> > git tree:       linux-next
> > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> >
> > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
>
> Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672

This commit was in the tested tree already as far as I can see.

Re: BUG: corrupted list in __dentry_kill (2)

[Al Viro]

On Thu, Dec 12, 2019 at 07:48:14AM +0100, Dmitry Vyukov wrote:
> On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
> >
> > On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > > Hello,
> > >
> > > syzbot found the following crash on:
> > >
> > > HEAD commit:    938f49c8 Add linux-next specific files for 20191211
> > > git tree:       linux-next
> > > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > > kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> > >
> > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
> >
> > Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
> 
> This commit was in the tested tree already as far as I can see.

Broken version (653f0d05be0948e7610bb786e6570bb6c48a4e75) is there, its
fixed replacement (a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672) is not.

Look, I realize that your setup is oriented to "followup commit Y fixes
a bug in earlier commit X", and sometimes it's the only possibility
(when X has already been in mainline), but in general it's spelled
"bisection hazard for no damn reason".  Fixes are folded in.
Routinely.  What's more, in this case the fixed version had been done
(and pushed out) before syzbot has seen the original, so putting
any metadata into commit message hadn't been an option.

If there is some format understandable for syzbot for such cases
("bug is caused by commit X; Y is a replacement that should not
exhibit the same bug, so if you see that behaviour on a tree
that doesn't contain X, report it.  X-containing trees ought
to go extinct reasonably soon"), please tell what it is.
Otherwise this situation will keep repeating - I am not going
to stop folding fixes into developing patches.

Speaking of bisect hazards, I'd recommend to check how your bisect
went - the bug is definitely local to this commit and I really
wonder what had caused the bisect to go wrong in this particular
case.

Re: BUG: corrupted list in __dentry_kill (2)

[Dmitry Vyukov]

On Thu, Dec 12, 2019 at 2:38 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Dec 12, 2019 at 07:48:14AM +0100, Dmitry Vyukov wrote:
> > On Thu, Dec 12, 2019 at 7:12 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
> > >
> > > On Wed, Dec 11, 2019 at 09:59:11PM -0800, syzbot wrote:
> > > > Hello,
> > > >
> > > > syzbot found the following crash on:
> > > >
> > > > HEAD commit:    938f49c8 Add linux-next specific files for 20191211
> > > > git tree:       linux-next
> > > > console output: https://syzkaller.appspot.com/x/log.txt?x=150eba1ee00000
> > > > kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
> > > > dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
> > > > compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
> > > > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
> > > > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000
> > > >
> > > > IMPORTANT: if you fix the bug, please add the following tag to the commit:
> > > > Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
> > >
> > > Already fixed in a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672
> >
> > This commit was in the tested tree already as far as I can see.
>
> Broken version (653f0d05be0948e7610bb786e6570bb6c48a4e75) is there, its
> fixed replacement (a3d1e7eb5abe3aa1095bc75d1a6760d3809bd672) is not.

Right, sorry, I looked only at the title when checked if it's in the
tree or not.

> Look, I realize that your setup is oriented to "followup commit Y fixes
> a bug in earlier commit X", and sometimes it's the only possibility
> (when X has already been in mainline), but in general it's spelled
> "bisection hazard for no damn reason".  Fixes are folded in.
> Routinely.  What's more, in this case the fixed version had been done
> (and pushed out) before syzbot has seen the original, so putting
> any metadata into commit message hadn't been an option.
>
> If there is some format understandable for syzbot for such cases
> ("bug is caused by commit X; Y is a replacement that should not
> exhibit the same bug, so if you see that behaviour on a tree
> that doesn't contain X, report it.  X-containing trees ought
> to go extinct reasonably soon"), please tell what it is.
> Otherwise this situation will keep repeating - I am not going
> to stop folding fixes into developing patches.

I did not find anything that could serve as these X/Y here. Commit
hashes in linux-next are pointless and won't match against other
trees. Commit titles seem to be the best bet for kernel, but this is
exactly the case where they fall short.

But in this case I think we can safely mark this as:

#syz fix:
simple_recursive_removal(): kernel-side rm -rf for ramfs-style filesystems

This should work because syzbot does coarser-grained check than "if
you see that behaviour on a tree that doesn't contain X". Instead it
will wait until _all_ tested trees will have X and only then close the
bug and report it again if it sees it again. Since the commit is
already fixed in linux-next, when all trees including mainline and
bpf/bpf-next will get it, it should be the fixed version.


> Speaking of bisect hazards, I'd recommend to check how your bisect
> went - the bug is definitely local to this commit and I really
> wonder what had caused the bisect to go wrong in this particular
> case.

I did not get the relation of folding to bisection. Or you mean these
are just separate things?
In this particular case the reason seems to be the same as for most
kernel bisections -- too many unrelated bugs layering one onto
another.
If you are interested in more details about why kernel bisection go
wrong in general, I've published a bunch of info here (including
detailed stats on reason):
https://groups.google.com/g/syzkaller/c/sR8aAXaWEF4/m/tTWYRgvmAwAJ
https://github.com/google/syzkaller/issues/1051
https://docs.google.com/spreadsheets/d/1WdBAN54-csaZpD3LgmTcIMR7NDFuQoOZZqPZ-CUqQgA/edit#gid=0

Re: BUG: corrupted list in __dentry_kill (2)

[Al Viro]

On Thu, Dec 12, 2019 at 04:57:14PM +0100, Dmitry Vyukov wrote:

> > Speaking of bisect hazards, I'd recommend to check how your bisect
> > went - the bug is definitely local to this commit and I really
> > wonder what had caused the bisect to go wrong in this particular
> > case.
> 
> I did not get the relation of folding to bisection. Or you mean these
> are just separate things?

Suppose instead of folding the fix in I would've done a followup commit
just with the fix.  And left the branch in that form, eventually getting
it pulled into mainline.  From that point on, *ANY* bisect stepping into
the first commit would've been thrown off.  For ever and ever, since
once it's in mainline, it really won't go away.

That's what folding avoids - accumulation of scar tissue, if you will.
Sure, there's enough cases when bug is found too late - it's already
in mainline or pulled into net-next or some other branch with similar
"no rebase, no reorder" policy.  But if you look at the patchsets posted
on the lists and watch them from iteration to iteration, you'll see
a _lot_ of fix-folding.  IME (both by my own practice and by watching
the patchsets posted by others) it outnumbers the cases when fix can't
be folded by quite a factor.  I wouldn't be surprised if it was an
order of magnitude...

Strict "never fold fixes" policy would've accelerated the accumulation
of bisect hazards in the mainline.  And while useful bisect may be a lost
cause for CI bots, it isn't that for intelligent developers.  Anything
that makes it more painful is not going to be welcome.

Re: BUG: corrupted list in __dentry_kill (2)

[Dmitry Vyukov]

On Thu, Dec 12, 2019 at 7:34 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Thu, Dec 12, 2019 at 04:57:14PM +0100, Dmitry Vyukov wrote:
>
> > > Speaking of bisect hazards, I'd recommend to check how your bisect
> > > went - the bug is definitely local to this commit and I really
> > > wonder what had caused the bisect to go wrong in this particular
> > > case.
> >
> > I did not get the relation of folding to bisection. Or you mean these
> > are just separate things?
>
> Suppose instead of folding the fix in I would've done a followup commit
> just with the fix.  And left the branch in that form, eventually getting
> it pulled into mainline.  From that point on, *ANY* bisect stepping into
> the first commit would've been thrown off.  For ever and ever, since
> once it's in mainline, it really won't go away.
>
> That's what folding avoids - accumulation of scar tissue, if you will.
> Sure, there's enough cases when bug is found too late - it's already
> in mainline or pulled into net-next or some other branch with similar
> "no rebase, no reorder" policy.  But if you look at the patchsets posted
> on the lists and watch them from iteration to iteration, you'll see
> a _lot_ of fix-folding.  IME (both by my own practice and by watching
> the patchsets posted by others) it outnumbers the cases when fix can't
> be folded by quite a factor.  I wouldn't be surprised if it was an
> order of magnitude...
>
> Strict "never fold fixes" policy would've accelerated the accumulation
> of bisect hazards in the mainline.  And while useful bisect may be a lost
> cause for CI bots, it isn't that for intelligent developers.  Anything
> that makes it more painful is not going to be welcome.


Ah, I see. Yes, folding will help future bisections. In fact, an
unfolded bug somewhere in kernel history is exactly what caused wrong
result on bisection for this bug.
Just in case, I did not propose to not do folding here (or anywhere
else as far as I remember). Handling of folded fixes is documented in
syzbot docs:
https://goo.gl/tpsmEJ#rebuilt-treesamended-patches

Re: BUG: corrupted list in __dentry_kill (2)

[syzbot]

syzbot has bisected this bug to:

commit ab92d68fc22f9afab480153bd82a20f6e2533769
Author: Taehee Yoo <ap420073@gmail.com>
Date:   Mon Oct 21 18:47:51 2019 +0000

     net: core: add generic lockdep keys

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=12d37cb6e00000
start commit:   938f49c8 Add linux-next specific files for 20191211
git tree:       linux-next
final crash:    https://syzkaller.appspot.com/x/report.txt?x=11d37cb6e00000
console output: https://syzkaller.appspot.com/x/log.txt?x=16d37cb6e00000
kernel config:  https://syzkaller.appspot.com/x/.config?x=96834c884ba7bb81
dashboard link: https://syzkaller.appspot.com/bug?extid=31043da7725b6ec210f1
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=12dc83dae00000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=16ac8396e00000

Reported-by: syzbot+31043da7725b6ec210f1@syzkaller.appspotmail.com
Fixes: ab92d68fc22f ("net: core: add generic lockdep keys")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Re: BUG: corrupted list in __dentry_kill (2)

[Al Viro]

On Thu, Dec 12, 2019 at 06:43:55PM +0800, Hillf Danton wrote:

> Unpin dentry no more than once as it's pinned only once.

The first time is on the entry into simple_recursive_removal(), by
+       struct dentry *this = dget(dentry);