From: Song Liu <songliubraving@fb.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Song Liu <liu.song.a23@gmail.com>,
Tariq Toukan <tariqt@mellanox.com>,
Dmitry Safonov <0x7f454c46@gmail.com>,
open list <linux-kernel@vger.kernel.org>,
Maor Gottlieb <maorg@mellanox.com>,
"Kernel Team" <Kernel-team@fb.com>
Subject: Re: WARNING and PANIC in irq_matrix_free
Date: Mon, 28 May 2018 18:34:57 +0000 [thread overview]
Message-ID: <09A22A95-6BBD-48EB-A2FE-42BF6244F751@fb.com> (raw)
In-Reply-To: <alpine.DEB.2.21.1805281251140.1578@nanos.tec.linutronix.de>
> On May 28, 2018, at 3:53 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Fri, 25 May 2018, Song Liu wrote:
>> On Wed, May 23, 2018 at 1:49 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
>>> On Wed, 23 May 2018, Tariq Toukan wrote:
>>>> I have your patch merged into my internal branch, it prints the following:
>>>>
>>>> [ 4898.226258] Trying to clear prev_vector: 0
>>>> [ 4898.226439] Trying to clear prev_vector: 0
>>>>
>>>> i.e. vector(0) is lower than FIRST_EXTERNAL_VECTOR.
>>>
>>> Could you please enable the vector and irq matrix trace points and capture
>>> the trace when this happens?
>
> Does the patch below fix it?
>
> Thanks,
>
> tglx
>
> 8<-------------------
> diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
> index bb6f7a2148d7..54af3d4884b1 100644
> --- a/arch/x86/kernel/apic/vector.c
> +++ b/arch/x86/kernel/apic/vector.c
> @@ -148,6 +148,7 @@ static void apic_update_vector(struct irq_data *irqd, unsigned int newvec,
> * prev_vector for this and the offlined target case.
> */
> apicd->prev_vector = 0;
> + apicd->move_in_progress = false;
> if (!apicd->vector || apicd->vector == MANAGED_IRQ_SHUTDOWN_VECTOR)
> goto setnew;
> /*
This doesn't fix the issue with bnxt. Here is a trace with this patch:
[ 569.222495] WARNING: CPU: 20 PID: 0 at kernel/irq/matrix.c:373 irq_matrix_free+0x32/0xd0
[ 569.238811] BUG: unable to handle kernel
[ 569.238811] NULL pointer dereference
[ 569.238812] at 0000000000000000
[ 569.238812] IP: bnxt_poll+0x163/0x830
[ 569.238812] PGD 0
[ 569.238812] P4D 0
[ 569.238813] Oops: 0002 [#1] SMP PTI
[ 569.238813] Modules linked in:
[ 569.238813] nfsv3
[ 569.238814] nfs
[ 569.238814] fscache
[ 569.238814] ip6table_raw
[ 569.238814] ip6table_mangle
[ 569.238815] iptable_raw
[ 569.238815] iptable_mangle
[ 569.238815] ip6table_filter
[ 569.238816] xt_NFLOG
[ 569.238816] xt_comment
[ 569.238816] xt_statistic
[ 569.238816] iptable_filter
[ 569.238817] nfnetlink_log
[ 569.238817] tcp_diag
[ 569.238817] inet_diag
[ 569.238817] sb_edac
[ 569.238818] x86_pkg_temp_thermal
[ 569.238818] intel_powerclamp
[ 569.238818] coretemp
[ 569.238818] kvm_intel
[ 569.238818] kvm
[ 569.238819] irqbypass
[ 569.238819] iTCO_wdt
[ 569.238819] iTCO_vendor_support
[ 569.238819] lpc_ich
[ 569.238819] efivars
[ 569.238820] mfd_core
[ 569.238820] i2c_i801
[ 569.238820] ipmi_si
[ 569.238820] ipmi_devintf
[ 569.238820] ipmi_msghandler
[ 569.238821] button
[ 569.238821] acpi_cpufreq
[ 569.238821] sch_fq_codel
[ 569.238821] nfsd
[ 569.238821] nfs_acl
[ 569.238822] lockd
[ 569.238822] auth_rpcgss
[ 569.238822] oid_registry
[ 569.238822] grace
[ 569.238822] sunrpc
[ 569.238823] fuse
[ 569.238823] loop
[ 569.238823] efivarfs
[ 569.238823] autofs4
[ 569.238824] CPU: 20 PID: 0 Comm: swapper/20 Not tainted 4.16.0-00391-g3742c6a #813
[ 569.238824] Hardware name: Quanta Leopard ORv2-DDR4/Leopard ORv2-DDR4, BIOS F06_3B12 08/17/2017
[ 569.238824] RIP: 0010:bnxt_poll+0x163/0x830
[ 569.238825] RSP: 0018:ffff883ffef83b18 EFLAGS: 00010006
[ 569.238825] RAX: 000000002c000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 569.238825] RDX: 0000000000000000 RSI: ffff881fdc53c000 RDI: ffff881f55a1da80
[ 569.238826] RBP: ffff883ffef83b60 R08: 0000000000000011 R09: 000000004fbf4c8d
[ 569.238826] R10: ffff883ffef83ab0 R11: ffff883fec6eee02 R12: ffff881ff6be2780
[ 569.238826] R13: 0000000000000000 R14: ffff881f55a1da80 R15: 0000000000000000
[ 569.238827] FS: 0000000000000000(0000) GS:ffff883ffef80000(0000) knlGS:0000000000000000
[ 569.238827] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 569.238827] CR2: 0000000000000000 CR3: 000000000220a003 CR4: 00000000003606e0
[ 569.238827] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 569.238828] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 569.238828] Call Trace:
[ 569.238828] <IRQ>
[ 569.238828] ? __skb_tx_hash+0x94/0xb0
[ 569.238829] netpoll_poll_dev+0xc5/0x1a0
[ 569.238829] netpoll_send_skb_on_dev+0x12c/0x200
[ 569.238829] netpoll_send_udp+0x2d5/0x410
[ 569.238829] write_ext_msg+0x1e7/0x200
[ 569.238830] ? scnprintf+0x3a/0x70
[ 569.238830] console_unlock+0x35c/0x530
[ 569.238830] vprintk_emit+0x225/0x340
[ 569.238830] ? irq_matrix_free+0x32/0xd0
[ 569.238831] vprintk_default+0x1f/0x30
[ 569.238831] vprintk_func+0x35/0x70
[ 569.238831] printk+0x43/0x4b
[ 569.238832] ? irq_matrix_free+0x32/0xd0
[ 569.238832] __warn+0x6f/0x150
[ 569.238832] ? irq_matrix_free+0x32/0xd0
[ 569.238832] report_bug+0x83/0xe0
[ 569.238833] do_invalid_op+0x2c/0x70
[ 569.238833] invalid_op+0x1b/0x40
[ 569.238833] RIP: 0010:irq_matrix_free+0x32/0xd0
[ 569.238834] RSP: 0018:ffff883ffef83f58 EFLAGS: 00010002
[ 569.238834] RAX: 0000000000000014 RBX: 0000000000014140 RCX: 0000000000000000
[ 569.238835] RDX: 0000000000000000 RSI: 0000000000000014 RDI: ffff881fff028c00
[ 569.238835] RBP: ffff883ffef83fc0 R08: ffff883ffefa1c40 R09: 0000000000000000
[ 569.238836] R10: 0000000000000000 R11: 0000000000000000 R12: ffff883ffefa4e00
[ 569.238836] R13: 0000000000000000 R14: 0000000000000014 R15: ffff881fff028c00
[ 569.238836] ? free_moved_vector+0x60/0x110
[ 569.238836] smp_irq_move_cleanup_interrupt+0x91/0xa9
[ 569.238837] irq_move_cleanup_interrupt+0xc/0x20
[ 569.238837] </IRQ>
[ 569.238837] RIP: 0010:cpuidle_enter_state+0xb6/0x2c0
[ 569.238837] RSP: 0018:ffffc9000c5d7e80 EFLAGS: 00000246
[ 569.238838] ORIG_RAX: ffffffffffffffdf
[ 569.238838] RAX: ffff883ffefa1080 RBX: ffffe8ffff780d00 RCX: 000000000000001f
[ 569.238838] RDX: 20c49ba5e353f7cf RSI: 0000000000000004 RDI: 0000000000000000
[ 569.238838] RBP: ffffc9000c5d7eb8 R08: fffa7b7805587a32 R09: ffffffff822ec140
[ 569.238839] R10: ffffc9000c5d7e60 R11: 0000000000004817 R12: 0000000000000004
[ 569.238839] R13: 0000000000000004 R14: 0000000000000014 R15: 0000008487066592
[ 569.238839] cpuidle_enter+0x17/0x20
[ 569.238839] call_cpuidle+0x2d/0x40
[ 569.238840] do_idle+0x109/0x1a0
[ 569.238840] cpu_startup_entry+0x1d/0x20
[ 569.238840] start_secondary+0x10e/0x120
[ 569.238840] secondary_startup_64+0xa5/0xb0
[ 569.238841] Code:
[ 569.238841] 89
[ 569.238841] 02
[ 569.238841] 41
[ 569.238841] f6
[ 569.238842] 44
[ 569.238842] 24
[ 569.238842] 36
[ 569.238842] 40
[ 569.238842] 74
[ 569.238842] 02
[ 569.238843] 89
[ 569.238843] 02
[ 569.238843] 8b
[ 569.238843] 44
[ 569.238843] 24
[ 569.238843] 1c
[ 569.238844] 49
[ 569.238844] 8b
[ 569.238844] 96
[ 569.238844] c8
[ 569.238844] 00
[ 569.238844] 00
[ 569.238845] 00
[ 569.238845] 41
[ 569.238845] 89
[ 569.238845] 86
[ 569.238845] c0
[ 569.238845] 00
[ 569.238846] 00
[ 569.238846] 00
[ 569.238846] 41
[ 569.238846] 23
[ 569.238846] 84
[ 569.238846] 24
[ 569.238847] d4
[ 569.238847] 00
[ 569.238847] 00
[ 569.238847] 00
[ 569.238847] 0d
[ 569.238847] 00
[ 569.238848] 00
[ 569.238848] 00
[ 569.238848] 2c
[ 569.238848] <89>
[ 569.238848] 02
[ 569.238848] 85
[ 569.238849] db
[ 569.238849] 0f
[ 569.238849] 85
[ 569.238849] e7
[ 569.238849] 00
[ 569.238849] 00
[ 569.238850] 00
[ 569.238850] f6
[ 569.238850] 44
[ 569.238850] 24
[ 569.238850] 1b
[ 569.238850] 01
[ 569.238851] 74
[ 569.238851] 40
[ 569.238851] 49
[ 569.238851] 8b
[ 569.238851] 96
[ 569.238851] a8
[ 569.238852] RIP: bnxt_poll+0x163/0x830 RSP: ffff883ffef83b18
[ 569.238852] CR2: 0000000000000000
next prev parent reply other threads:[~2018-05-28 18:36 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-20 12:07 Tariq Toukan
2018-02-20 18:11 ` Thomas Gleixner
2018-02-20 18:18 ` Thomas Gleixner
2018-02-21 11:27 ` Tariq Toukan
2018-02-22 21:38 ` Thomas Gleixner
2018-02-25 9:50 ` Tariq Toukan
2018-05-18 22:41 ` Dmitry Safonov
2018-05-18 23:43 ` Dmitry Safonov
2018-05-19 11:20 ` Thomas Gleixner
2018-05-23 7:16 ` Tariq Toukan
2018-05-23 8:49 ` Thomas Gleixner
2018-05-25 20:10 ` Song Liu
2018-05-25 21:29 ` Song Liu
2018-05-28 10:53 ` Thomas Gleixner
2018-05-28 11:17 ` Tariq Toukan
2018-05-28 14:27 ` Thomas Gleixner
2018-05-28 18:36 ` Song Liu
2018-05-28 18:34 ` Song Liu [this message]
2018-05-28 20:09 ` Thomas Gleixner
[not found] ` <3F47F523-64C5-422B-B9B0-73B8D105CF71@fb.com>
2018-05-29 8:35 ` Thomas Gleixner
2018-05-29 16:54 ` Song Liu
2018-05-30 21:56 ` Thomas Gleixner
2018-06-04 7:56 ` Dou Liyang
2018-06-04 11:17 ` Thomas Gleixner
2018-06-04 11:59 ` Dou Liyang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=09A22A95-6BBD-48EB-A2FE-42BF6244F751@fb.com \
--to=songliubraving@fb.com \
--cc=0x7f454c46@gmail.com \
--cc=Kernel-team@fb.com \
--cc=linux-kernel@vger.kernel.org \
--cc=liu.song.a23@gmail.com \
--cc=maorg@mellanox.com \
--cc=tariqt@mellanox.com \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®