* Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) @ 2016-09-16 2:20 Guenter Roeck 2016-09-16 6:45 ` Jan Kara 0 siblings, 1 reply; 8+ messages in thread From: Guenter Roeck @ 2016-09-16 2:20 UTC (permalink / raw) To: linux-kernel, Jan Kara, Andrew Morton Hi, I see various architectures crashing in -next with the following error. ------------[ cut here ]------------ kernel BUG at fs/notify/notification.c:66! invalid opcode: 0000 [#1] PREEMPT Modules linked in: CPU: 0 PID: 110 Comm: udevd Not tainted 4.8.0-rc6-next-20160915-yocto-standard #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014 task: ffff88000dd58000 task.stack: ffffc90000250000 RIP: 0010:[<ffffffff811bb399>] [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10 RSP: 0018:ffffc90000253e68 EFLAGS: 00000296 RAX: ffff88000dd58000 RBX: ffff88000dc9a9c0 RCX: 0000000000011ff0 RDX: ffffffff811bd52a RSI: ffff88000e3c1258 RDI: ffff88000dc9a9c0 RBP: ffffc90000253e68 R08: ffff88000ffd37e0 R09: ffff88000dcdd600 R10: ffff88000dcdd600 R11: 0000000000000000 R12: 0000000000000001 R13: ffff88000dd28c00 R14: ffff88000dc0a380 R15: ffff88000dc0a398 FS: 00007ff5bea85740(0000) GS:ffffffff81c31000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff5bea79000 CR3: 000000000dd63000 CR4: 00000000000406f0 Stack: ffffc90000253e88 ffffffff811bd532 0000000000000000 ffff88000dd28c00 ffffc90000253f48 ffffffff811bfeba ffffffff00000006 0000000000000000 ffff88000dcdd600 ffff88000dcdd600 0000001900253f58 0000040000080002 Call Trace: [<ffffffff811bd532>] inotify_poll+0x42/0x70 [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60 [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180 [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48 RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10 RSP <ffffc90000253e68> ---[ end trace 7dc4a27003f0b575 ]--- I didn't bisect, but I would guess the culprit is one of the new patches in the affected file. 22e9cf146d3b fanotify: fix possible false warning when freeing events ced89591817c fsnotify: convert notification_mutex to a spinlock f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response() 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown Guenter ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-16 2:20 Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck @ 2016-09-16 6:45 ` Jan Kara 2016-09-16 11:10 ` Guenter Roeck 0 siblings, 1 reply; 8+ messages in thread From: Jan Kara @ 2016-09-16 6:45 UTC (permalink / raw) To: Guenter Roeck; +Cc: linux-kernel, Jan Kara, Andrew Morton Hi, On Thu 15-09-16 19:20:10, Guenter Roeck wrote: > I see various architectures crashing in -next with the following error. > > ------------[ cut here ]------------ > kernel BUG at fs/notify/notification.c:66! Thanks for report! ... > Call Trace: > [<ffffffff811bd532>] inotify_poll+0x42/0x70 > [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60 > [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180 > [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f > Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48 > RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10 > RSP <ffffc90000253e68> > ---[ end trace 7dc4a27003f0b575 ]--- > > I didn't bisect, but I would guess the culprit is one of the new patches in the > affected file. > > 22e9cf146d3b fanotify: fix possible false warning when freeing events > ced89591817c fsnotify: convert notification_mutex to a spinlock > f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event > 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response() > 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown Yes, very likely my patches cause this. But it must be some config issue because my test machine does not hit this and the code is "obviously correct" - famous last words ;). I've seen zero-day robot complain as well so I'll try the config it had sent me and see what's going on (likely spin_is_locked() does not do what I thought under some configs...). Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-16 6:45 ` Jan Kara @ 2016-09-16 11:10 ` Guenter Roeck 2016-09-16 12:49 ` Jan Kara 0 siblings, 1 reply; 8+ messages in thread From: Guenter Roeck @ 2016-09-16 11:10 UTC (permalink / raw) To: Jan Kara; +Cc: linux-kernel, Andrew Morton On 09/15/2016 11:45 PM, Jan Kara wrote: > Hi, > > On Thu 15-09-16 19:20:10, Guenter Roeck wrote: >> I see various architectures crashing in -next with the following error. >> >> ------------[ cut here ]------------ >> kernel BUG at fs/notify/notification.c:66! > > Thanks for report! > > ... > >> Call Trace: >> [<ffffffff811bd532>] inotify_poll+0x42/0x70 >> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60 >> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180 >> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f >> Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48 >> RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10 >> RSP <ffffc90000253e68> >> ---[ end trace 7dc4a27003f0b575 ]--- >> >> I didn't bisect, but I would guess the culprit is one of the new patches in the >> affected file. >> >> 22e9cf146d3b fanotify: fix possible false warning when freeing events >> ced89591817c fsnotify: convert notification_mutex to a spinlock >> f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event >> 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response() >> 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown > > Yes, very likely my patches cause this. But it must be some config issue > because my test machine does not hit this and the code is "obviously > correct" - famous last words ;). I've seen zero-day robot complain as well > so I'll try the config it had sent me and see what's going on (likely > spin_is_locked() does not do what I thought under some configs...). > Your builds are probably all SMP builds. Looks like this happens if SMP is disabled. Guenter ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-16 11:10 ` Guenter Roeck @ 2016-09-16 12:49 ` Jan Kara 2016-09-20 1:10 ` Michael Ellerman 0 siblings, 1 reply; 8+ messages in thread From: Jan Kara @ 2016-09-16 12:49 UTC (permalink / raw) To: Guenter Roeck; +Cc: Jan Kara, linux-kernel, Andrew Morton On Fri 16-09-16 04:10:38, Guenter Roeck wrote: > On 09/15/2016 11:45 PM, Jan Kara wrote: > >Hi, > > > >On Thu 15-09-16 19:20:10, Guenter Roeck wrote: > >>I see various architectures crashing in -next with the following error. > >> > >>------------[ cut here ]------------ > >>kernel BUG at fs/notify/notification.c:66! > > > >Thanks for report! > > > >... > > > >>Call Trace: > >> [<ffffffff811bd532>] inotify_poll+0x42/0x70 > >> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60 > >> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180 > >> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f > >>Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48 > >>RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10 > >> RSP <ffffc90000253e68> > >>---[ end trace 7dc4a27003f0b575 ]--- > >> > >>I didn't bisect, but I would guess the culprit is one of the new patches in the > >>affected file. > >> > >>22e9cf146d3b fanotify: fix possible false warning when freeing events > >>ced89591817c fsnotify: convert notification_mutex to a spinlock > >>f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event > >>782fbc7e8685 fanotify: fix list corruption in fanotify_get_response() > >>56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown > > > >Yes, very likely my patches cause this. But it must be some config issue > >because my test machine does not hit this and the code is "obviously > >correct" - famous last words ;). I've seen zero-day robot complain as well > >so I'll try the config it had sent me and see what's going on (likely > >spin_is_locked() does not do what I thought under some configs...). > > > > Your builds are probably all SMP builds. Looks like this happens if SMP > is disabled. Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG returns always 0. Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-16 12:49 ` Jan Kara @ 2016-09-20 1:10 ` Michael Ellerman 2016-09-20 1:13 ` Andrew Morton 0 siblings, 1 reply; 8+ messages in thread From: Michael Ellerman @ 2016-09-20 1:10 UTC (permalink / raw) To: Jan Kara, Guenter Roeck; +Cc: linux-kernel, Andrew Morton Jan Kara <jack@suse.cz> writes: > On Fri 16-09-16 04:10:38, Guenter Roeck wrote: >> On 09/15/2016 11:45 PM, Jan Kara wrote: >> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote: >> >>I see various architectures crashing in -next with the following error. >> >>------------[ cut here ]------------ >> >>kernel BUG at fs/notify/notification.c:66! >> >> >correct" - famous last words ;). I've seen zero-day robot complain as well >> >so I'll try the config it had sent me and see what's going on (likely >> >spin_is_locked() does not do what I thought under some configs...). >> >> Your builds are probably all SMP builds. Looks like this happens if SMP >> is disabled. > > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG > returns always 0. Can we get this fixed soon please? It's breaking all my CI runs. cheers ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-20 1:10 ` Michael Ellerman @ 2016-09-20 1:13 ` Andrew Morton 2016-09-20 12:05 ` Michael Ellerman 0 siblings, 1 reply; 8+ messages in thread From: Andrew Morton @ 2016-09-20 1:13 UTC (permalink / raw) To: Michael Ellerman; +Cc: Jan Kara, Guenter Roeck, linux-kernel On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote: > Jan Kara <jack@suse.cz> writes: > > On Fri 16-09-16 04:10:38, Guenter Roeck wrote: > >> On 09/15/2016 11:45 PM, Jan Kara wrote: > >> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote: > >> >>I see various architectures crashing in -next with the following error. > >> >>------------[ cut here ]------------ > >> >>kernel BUG at fs/notify/notification.c:66! > >> > >> >correct" - famous last words ;). I've seen zero-day robot complain as well > >> >so I'll try the config it had sent me and see what's going on (likely > >> >spin_is_locked() does not do what I thought under some configs...). > >> > >> Your builds are probably all SMP builds. Looks like this happens if SMP > >> is disabled. > > > > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG > > returns always 0. > > Can we get this fixed soon please? It's breaking all my CI runs. > It should be fixed in next -next. From: Jan Kara <jack@suse.cz> Subject: fsnotify-convert-notification_mutex-to-a-spinlock-fix This is a fixed version of the patch that fixes the BUG_ON hitting on UP kernels. Link: http://lkml.kernel.org/r/1474031567-1831-1-git-send-email-jack@suse.cz Signed-off-by: Jan Kara <jack@suse.cz> Reviewed-by: Lino Sanfilippo <LinoSanfilippo@gmx.de> Tested-by: Guenter Roeck <linux@roeck-us.net> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> --- fs/notify/fanotify/fanotify_user.c | 3 ++- fs/notify/notification.c | 9 ++++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix +++ a/fs/notify/fanotify/fanotify_user.c @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c static struct fsnotify_event *get_one_event(struct fsnotify_group *group, size_t count) { - BUG_ON(!spin_is_locked(&group->notification_lock)); + BUG_ON(IS_ENABLED(CONFIG_SMP) && + !spin_is_locked(&group->notification_lock)); pr_debug("%s: group=%p count=%zd\n", __func__, group, count); diff -puN fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/notification.c --- a/fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix +++ a/fs/notify/notification.c @@ -63,7 +63,8 @@ EXPORT_SYMBOL_GPL(fsnotify_get_cookie); /* return true if the notify queue is empty, false otherwise */ bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group) { - BUG_ON(!spin_is_locked(&group->notification_lock)); + BUG_ON(IS_ENABLED(CONFIG_SMP) && + !spin_is_locked(&group->notification_lock)); return list_empty(&group->notification_list) ? true : false; } @@ -139,7 +140,8 @@ struct fsnotify_event *fsnotify_remove_f { struct fsnotify_event *event; - BUG_ON(!spin_is_locked(&group->notification_lock)); + BUG_ON(IS_ENABLED(CONFIG_SMP) && + !spin_is_locked(&group->notification_lock)); pr_debug("%s: group=%p\n", __func__, group); @@ -161,7 +163,8 @@ struct fsnotify_event *fsnotify_remove_f */ struct fsnotify_event *fsnotify_peek_first_event(struct fsnotify_group *group) { - BUG_ON(!spin_is_locked(&group->notification_lock)); + BUG_ON(IS_ENABLED(CONFIG_SMP) && + !spin_is_locked(&group->notification_lock)); return list_first_entry(&group->notification_list, struct fsnotify_event, list); _ ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-20 1:13 ` Andrew Morton @ 2016-09-20 12:05 ` Michael Ellerman 2016-09-21 9:15 ` Jan Kara 0 siblings, 1 reply; 8+ messages in thread From: Michael Ellerman @ 2016-09-20 12:05 UTC (permalink / raw) To: Andrew Morton; +Cc: Jan Kara, Guenter Roeck, linux-kernel Andrew Morton <akpm@linux-foundation.org> writes: > On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote: >> Jan Kara <jack@suse.cz> writes: >> > >> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG >> > returns always 0. >> >> Can we get this fixed soon please? It's breaking all my CI runs. > > It should be fixed in next -next. Great thanks. I did search LKML to see if Jan had sent a fix but I guess I missed it. > diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c > --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix > +++ a/fs/notify/fanotify/fanotify_user.c > @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c > static struct fsnotify_event *get_one_event(struct fsnotify_group *group, > size_t count) > { > - BUG_ON(!spin_is_locked(&group->notification_lock)); > + BUG_ON(IS_ENABLED(CONFIG_SMP) && > + !spin_is_locked(&group->notification_lock)); I thought lockdep_assert_held() was preferred for checks like this that are purely sanity checking, ie. not part of the algorithm. cheers ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) 2016-09-20 12:05 ` Michael Ellerman @ 2016-09-21 9:15 ` Jan Kara 0 siblings, 0 replies; 8+ messages in thread From: Jan Kara @ 2016-09-21 9:15 UTC (permalink / raw) To: Michael Ellerman; +Cc: Andrew Morton, Jan Kara, Guenter Roeck, linux-kernel On Tue 20-09-16 22:05:00, Michael Ellerman wrote: > Andrew Morton <akpm@linux-foundation.org> writes: > > On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote: > >> Jan Kara <jack@suse.cz> writes: > >> > > >> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG > >> > returns always 0. > >> > >> Can we get this fixed soon please? It's breaking all my CI runs. > > > > It should be fixed in next -next. > > Great thanks. > > I did search LKML to see if Jan had sent a fix but I guess I missed it. I didn't send the fix to LKML but only to linux-fsdevel... > > diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c > > --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix > > +++ a/fs/notify/fanotify/fanotify_user.c > > @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c > > static struct fsnotify_event *get_one_event(struct fsnotify_group *group, > > size_t count) > > { > > - BUG_ON(!spin_is_locked(&group->notification_lock)); > > + BUG_ON(IS_ENABLED(CONFIG_SMP) && > > + !spin_is_locked(&group->notification_lock)); > > I thought lockdep_assert_held() was preferred for checks like this that > are purely sanity checking, ie. not part of the algorithm. As pointed out by other guy, assert_spin_locked() is probably the right way to do it (it's a wrapper that does essentially what I did here). Somehow I missed that when looking for the right fix. Honza -- Jan Kara <jack@suse.com> SUSE Labs, CR ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-09-21 9:15 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-09-16 2:20 Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck 2016-09-16 6:45 ` Jan Kara 2016-09-16 11:10 ` Guenter Roeck 2016-09-16 12:49 ` Jan Kara 2016-09-20 1:10 ` Michael Ellerman 2016-09-20 1:13 ` Andrew Morton 2016-09-20 12:05 ` Michael Ellerman 2016-09-21 9:15 ` Jan Kara
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®