Hi all, I'm seeing a system crash while running perf_fuzzer with upstream kernel on an Intel machine. I hit the crash twice (out of which I don't have log of first crash so don't know if the reason is same for both the crashes). I've attached my .config with the mail. System configuration: ======== $ uname -r 5.0.0-rc3-ravi-pfuzzer+ $ gcc --version gcc (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010 $ cat /etc/os-release NAME="Ubuntu" VERSION="18.04.1 LTS (Bionic Beaver)" $ lscpu CPU(s): 64 On-line CPU(s) list: 0-3 (<===== I kept only 4 cpus ON) Off-line CPU(s) list: 4-63 Model name: Intel(R) Xeon(R) CPU E5-4640 0 @ 2.40GHz ======== And I fuzz only Hardware pmu events. Also, I disabled /proc/sys/kernel/perf* files manipulation by perf_fuzzer: ======== $ git diff diff --git a/fuzzer/perf_fuzzer.c b/fuzzer/perf_fuzzer.c index 7110fd6..6dcb307 100644 --- a/fuzzer/perf_fuzzer.c +++ b/fuzzer/perf_fuzzer.c @@ -711,7 +711,7 @@ int main(int argc, char **argv) { } break; case 6: if (type&TYPE_ACCESS) { - access_random_file(); + //access_random_file(); } break; case 7: if (type&TYPE_FORK) { diff --git a/fuzzer/trinity_files/perf_event_open.c b/fuzzer/trinity_files/perf_event_open.c index 2ef3739..d5f4948 100644 --- a/fuzzer/trinity_files/perf_event_open.c +++ b/fuzzer/trinity_files/perf_event_open.c @@ -615,6 +615,8 @@ static int random_event_type(void) int type=0; + return PERF_TYPE_HARDWARE; + switch (rand() % 8) { case 0: type = PERF_TYPE_HARDWARE; ======== And, I'm running fuzzer in a loop with *root*. (Let me know if running as root is harmful ;-) ). ======== # while true; do ./perf_fuzzer; done ======== I started fuzzer at: Fri Jan 25 10:28:44 IST 2019 dmesg till system was running nornal: [Fri Jan 25 10:28:53 2019] perf: interrupt took too long (2501 > 2500), lowering kernel.perf_event_max_sample_rate to 79750 [Fri Jan 25 10:29:08 2019] perf: interrupt took too long (3136 > 3126), lowering kernel.perf_event_max_sample_rate to 63750 [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (4140 > 3920), lowering kernel.perf_event_max_sample_rate to 48250 [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (5231 > 5175), lowering kernel.perf_event_max_sample_rate to 38000 [Fri Jan 25 10:29:11 2019] perf: interrupt took too long (6736 > 6538), lowering kernel.perf_event_max_sample_rate to 29500 [Fri Jan 25 10:32:44 2019] ------------[ cut here ]------------ [Fri Jan 25 10:32:44 2019] perfevents: irq loop stuck! [Fri Jan 25 10:32:44 2019] WARNING: CPU: 1 PID: 0 at arch/x86/events/intel/core.c:2440 intel_pmu_handle_irq+0x158/0x170 [Fri Jan 25 10:32:44 2019] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif input_leds kvm_intel cdc_ether usbnet kvm mii irqbypass sch_fq_codel ipmi_si ib_iser ioatdma ipmi_devintf rdma_cm ipmi_msghandler lpc_ich dca wmi mac_hid iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 multipath linear mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper be2net ahci megaraid_sas libahci hid_generic usbhid hid [Fri Jan 25 10:32:44 2019] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.0.0-rc3-ravi-pfuzzer+ #1 [Fri Jan 25 10:32:44 2019] Hardware name: IBM CPU PLANAR -[8722xyz]-/00FL808, BIOS -[KOE162DUS-2.30]- 08/27/2018 [Fri Jan 25 10:32:44 2019] RIP: 0010:intel_pmu_handle_irq+0x158/0x170 [Fri Jan 25 10:32:44 2019] Code: 10 e8 4c 36 bf 00 e9 06 ff ff ff 80 3d b0 07 8a 01 00 74 0a e8 f9 df ff ff e9 78 ff ff ff 48 c7 c7 82 cc ca 8c e8 08 d7 07 00 <0f> 0b e8 11 a2 ff ff c6 05 8a 07 8a 01 01 eb da 0f 1f 84 00 00 00 [Fri Jan 25 10:32:44 2019] RSP: 0000:fffffe0000032df8 EFLAGS: 00010082 [Fri Jan 25 10:32:44 2019] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000002 [Fri Jan 25 10:32:44 2019] RDX: 0000000000000003 RSI: ffffffff8ccacc9e RDI: 0000000000000000 [Fri Jan 25 10:32:44 2019] RBP: fffffe0000032e28 R08: 0000000000000000 R09: 0000000000000000 [Fri Jan 25 10:32:44 2019] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000002 [Fri Jan 25 10:32:44 2019] R13: 0000000000000000 R14: fffffe0000032ef8 R15: ffff95b3ff84f3a0 [Fri Jan 25 10:32:44 2019] FS: 0000000000000000(0000) GS:ffff95b3ff840000(0000) knlGS:0000000000000000 [Fri Jan 25 10:32:44 2019] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Fri Jan 25 10:32:44 2019] CR2: 00007ffdc4de0238 CR3: 0000001fd220e003 CR4: 00000000000606e0 [Fri Jan 25 10:32:44 2019] Call Trace: [Fri Jan 25 10:32:44 2019] [Fri Jan 25 10:32:44 2019] perf_event_nmi_handler+0x33/0x50 [Fri Jan 25 10:32:44 2019] nmi_handle+0x6e/0x120 [Fri Jan 25 10:32:44 2019] default_do_nmi+0x44/0x120 [Fri Jan 25 10:32:44 2019] do_nmi+0x119/0x180 [Fri Jan 25 10:32:44 2019] end_repeat_nmi+0x16/0x50 [Fri Jan 25 10:32:44 2019] RIP: 0010:intel_pmu_lbr_enable_all+0x2d/0xf0 [Fri Jan 25 10:32:44 2019] Code: 66 90 55 48 c7 c2 a0 f3 00 00 48 89 d0 48 89 e5 41 54 53 65 48 03 05 5a e0 3f 74 8b 80 58 09 00 00 85 c0 75 05 5b 41 5c 5d c3 <65> 48 03 15 43 e0 3f 74 40 84 ff 75 7c 48 8b 82 68 0c 00 00 48 85 [Fri Jan 25 10:32:44 2019] RSP: 0000:ffff95b3ff843e48 EFLAGS: 00000002 [Fri Jan 25 10:32:44 2019] RAX: 0000000000000001 RBX: ffff95b3ff84f3a0 RCX: 00000000000003f1 [Fri Jan 25 10:32:44 2019] RDX: 000000000000f3a0 RSI: 0000000000000003 RDI: 0000000000000000 [Fri Jan 25 10:32:44 2019] RBP: ffff95b3ff843e58 R08: fffffe0000036000 R09: 0000000000000005 [Fri Jan 25 10:32:44 2019] R10: ffff95b3ff84fac0 R11: 0000000000000005 R12: 0000000000000000 [Fri Jan 25 10:32:44 2019] R13: ffff95b3ff84f3a0 R14: ffff95b3ff84f5d4 R15: 0000000000000005 [Fri Jan 25 10:32:44 2019] ? intel_pmu_lbr_enable_all+0x2d/0xf0 [Fri Jan 25 10:32:44 2019] ? intel_pmu_lbr_enable_all+0x2d/0xf0 [Fri Jan 25 10:32:44 2019] [Fri Jan 25 10:32:44 2019] [Fri Jan 25 10:32:44 2019] __intel_pmu_enable_all+0x2c/0x80 [Fri Jan 25 10:32:44 2019] intel_pmu_enable_all+0x10/0x20 [Fri Jan 25 10:32:44 2019] x86_pmu_enable+0x27f/0x310 [Fri Jan 25 10:32:44 2019] perf_pmu_enable.part.100+0xd/0x10 [Fri Jan 25 10:32:44 2019] ctx_resched+0x8f/0xd0 [Fri Jan 25 10:32:44 2019] __perf_event_enable+0x193/0x1e0 [Fri Jan 25 10:32:44 2019] event_function+0x8e/0xc0 [Fri Jan 25 10:32:44 2019] remote_function+0x41/0x50 [Fri Jan 25 10:32:44 2019] flush_smp_call_function_queue+0x68/0x100 [Fri Jan 25 10:32:44 2019] generic_smp_call_function_single_interrupt+0x13/0x30 [Fri Jan 25 10:32:44 2019] smp_call_function_single_interrupt+0x3e/0xe0 [Fri Jan 25 10:32:44 2019] call_function_single_interrupt+0xf/0x20 [Fri Jan 25 10:32:44 2019] [Fri Jan 25 10:32:44 2019] RIP: 0010:cpuidle_enter_state+0xc2/0x460 [Fri Jan 25 10:32:44 2019] Code: 66 90 31 ff e8 2f 19 8a ff 80 7d c7 00 74 17 9c 58 66 66 90 66 90 f6 c4 02 0f 85 69 03 00 00 31 ff e8 f2 25 90 ff fb 66 66 90 <66> 66 90 45 85 ed 0f 88 37 03 00 00 4c 8b 45 d0 4c 2b 45 b8 48 ba [Fri Jan 25 10:32:44 2019] RSP: 0000:ffffb1d70c527e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [Fri Jan 25 10:32:44 2019] RAX: ffff95b3ff862a00 RBX: ffffffff8d1449c0 RCX: 000000000000001f [Fri Jan 25 10:32:44 2019] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [Fri Jan 25 10:32:44 2019] RBP: ffffb1d70c527e90 R08: ffffd016493f3b42 R09: ffff95b3ff861a64 [Fri Jan 25 10:32:44 2019] R10: ffffb1d70c527e18 R11: 0000000000000034 R12: ffffd1aaefa40090 [Fri Jan 25 10:32:44 2019] R13: 0000000000000005 R14: 0000000000000005 R15: ffffffff8d1449c0 [Fri Jan 25 10:32:44 2019] ? cpuidle_enter_state+0xa1/0x460 [Fri Jan 25 10:32:44 2019] cpuidle_enter+0x17/0x20 [Fri Jan 25 10:32:44 2019] call_cpuidle+0x23/0x40 [Fri Jan 25 10:32:44 2019] do_idle+0x201/0x280 [Fri Jan 25 10:32:44 2019] cpu_startup_entry+0x1d/0x20 [Fri Jan 25 10:32:44 2019] start_secondary+0x1ab/0x200 [Fri Jan 25 10:32:44 2019] secondary_startup_64+0xa4/0xb0 [Fri Jan 25 10:32:44 2019] ---[ end trace 826a05dfaf1c2406 ]--- [Fri Jan 25 10:32:44 2019] CPU#1: ctrl: 0000000000000000 [Fri Jan 25 10:32:44 2019] CPU#1: status: 0000000000000002 [Fri Jan 25 10:32:44 2019] CPU#1: overflow: 0000000000000000 [Fri Jan 25 10:32:44 2019] CPU#1: fixed: 0000000000000bb0 [Fri Jan 25 10:32:44 2019] CPU#1: pebs: 0000000000000000 [Fri Jan 25 10:32:44 2019] CPU#1: debugctl: 0000000000000000 [Fri Jan 25 10:32:44 2019] CPU#1: active: 0000000600000007 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC0 ctrl: 0000000000410300 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC0 count: 0000800000000001 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC0 left: 00007fffffffffff [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC1 ctrl: 00000000004200c5 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC1 count: 0000ffffffffffff [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC1 left: 0000000000000002 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC2 ctrl: 000000000053003c [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC2 count: 000080000014f025 [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC2 left: 00007fffffeb0fdb [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC3 ctrl: 000000000010412e [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC3 count: 0000fffffffffffe [Fri Jan 25 10:32:44 2019] CPU#1: gen-PMC3 left: 0000000000000002 [Fri Jan 25 10:32:44 2019] CPU#1: fixed-PMC0 count: 0000800000000001 [Fri Jan 25 10:32:44 2019] CPU#1: fixed-PMC1 count: 0000ffffbf106c64 [Fri Jan 25 10:32:44 2019] CPU#1: fixed-PMC2 count: 000080000013cae9 [Fri Jan 25 10:32:44 2019] core: clearing PMU state on CPU#1 [Fri Jan 25 10:33:02 2019] perf: interrupt took too long (8592 > 8420), lowering kernel.perf_event_max_sample_rate to 23250 [Fri Jan 25 10:33:02 2019] perf: interrupt took too long (10793 > 10740), lowering kernel.perf_event_max_sample_rate to 18500 [Fri Jan 25 10:33:02 2019] perf: interrupt took too long (13764 > 13491), lowering kernel.perf_event_max_sample_rate to 14500 [Fri Jan 25 10:41:01 2019] perf: interrupt took too long (17219 > 17205), lowering kernel.perf_event_max_sample_rate to 11500 [Fri Jan 25 10:45:01 2019] perf: interrupt took too long (21567 > 21523), lowering kernel.perf_event_max_sample_rate to 9250 [Fri Jan 25 10:45:01 2019] perf: interrupt took too long (27058 > 26958), lowering kernel.perf_event_max_sample_rate to 7250 [Fri Jan 25 10:45:01 2019] perf: interrupt took too long (34097 > 33822), lowering kernel.perf_event_max_sample_rate to 5750 [Fri Jan 25 10:45:01 2019] perf: interrupt took too long (42633 > 42621), lowering kernel.perf_event_max_sample_rate to 4500 [Fri Jan 25 10:45:01 2019] perf: interrupt took too long (53320 > 53291), lowering kernel.perf_event_max_sample_rate to 3750 [Fri Jan 25 10:47:45 2019] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 3.193 msecs [Fri Jan 25 10:47:48 2019] perf: interrupt took too long (76120 > 66650), lowering kernel.perf_event_max_sample_rate to 2500 [Fri Jan 25 10:47:49 2019] perf: interrupt took too long (105206 > 95150), lowering kernel.perf_event_max_sample_rate to 1750 [Fri Jan 25 10:47:49 2019] perf: interrupt took too long (132931 > 131507), lowering kernel.perf_event_max_sample_rate to 1500 dmesg from console when system crashed: [ 1432.176374] general protection fault: 0000 [#1] SMP PTI [ 1432.182253] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 5.0.0-rc3-ravi-pfuzzer+ #1 [ 1432.192088] Hardware name: IBM CPU PLANAR -[8722xyz]-/00FL808, BIOS -[KOE162DUS-2.30]- 08/27/2018 [264/488] [ 1432.206120] RIP: 0010:perf_prepare_sample+0x8f/0x510 [ 1432.211679] Code: ff ff 41 f6 c4 01 0f 85 22 02 00 00 41 f6 c4 20 74 26 4d 85 e4 0f 88 0c 01 00 00 4c 89 f6 4c 89 ff e8 f5 fe ff ff 49 89 45 70 <48> 8b 00 8d 04 c5 08 00 00 0 0 66 01 43 06 41 f7 c4 00 04 00 00 74 [ 1432.232699] RSP: 0000:ffff95b3ff843a78 EFLAGS: 00010082 [ 1432.238548] RAX: 8d1217eb826cce00 RBX: ffff95b3ff843ad8 RCX: 000000000000001f [ 1432.246536] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1432.254522] RBP: ffff95b3ff843ab0 R08: ffffd016493f3b42 R09: 0000000000000000 [ 1432.262508] R10: ffff95b3ff843a08 R11: 0000000000000000 R12: 80000000000306e5 [ 1432.270495] R13: ffff95b3ff843bc0 R14: ffff95b3ff843b18 R15: ffff95b3f6b65800 [ 1432.278483] FS: 0000000000000000(0000) GS:ffff95b3ff840000(0000) knlGS:0000000000000000 [ 1432.287539] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1432.293969] CR2: 000055bf7f768c90 CR3: 0000001fd220e005 CR4: 00000000000606e0 [ 1432.301956] Call Trace: [ 1432.304697] [ 1432.306956] ? intel_pmu_drain_bts_buffer+0x194/0x230 [ 1432.312615] intel_pmu_drain_bts_buffer+0x160/0x230 [ 1432.318078] ? tick_nohz_irq_exit+0x31/0x40 [ 1432.322765] ? smp_call_function_single_interrupt+0x48/0xe0 [ 1432.328993] ? call_function_single_interrupt+0xf/0x20 [ 1432.334745] ? call_function_single_interrupt+0xa/0x20 [ 1432.340501] ? x86_schedule_events+0x1a0/0x2f0 [ 1432.345475] ? x86_pmu_commit_txn+0xb4/0x100 [ 1432.350258] ? find_busiest_group+0x47/0x5d0 [ 1432.355039] ? perf_event_set_state.part.42+0x12/0x50 [ 1432.360694] ? perf_mux_hrtimer_restart+0x40/0xb0 [ 1432.365960] intel_pmu_disable_event+0xae/0x100 [ 1432.371031] ? intel_pmu_disable_event+0xae/0x100 [ 1432.376297] x86_pmu_stop+0x7a/0xb0 [ 1432.380201] x86_pmu_del+0x57/0x120 [ 1432.384105] event_sched_out.isra.101+0x83/0x180 [ 1432.389272] group_sched_out.part.103+0x57/0xe0 [ 1432.394343] ctx_sched_out+0x188/0x240 [ 1432.398539] ctx_resched+0xa8/0xd0 [ 1432.402345] __perf_event_enable+0x193/0x1e0 [ 1432.407125] event_function+0x8e/0xc0 [ 1432.411222] remote_function+0x41/0x50 [ 1432.415418] flush_smp_call_function_queue+0x68/0x100 [ 1432.421071] generic_smp_call_function_single_interrupt+0x13/0x30 [ 1432.427893] smp_call_function_single_interrupt+0x3e/0xe0 [ 1432.433936] call_function_single_interrupt+0xf/0x20 [ 1432.440204] [ 1432.443267] RIP: 0010:cpuidle_enter_state+0xc2/0x460 [ 1432.449536] Code: 66 90 31 ff e8 2f 19 8a ff 80 7d c7 00 74 17 9c 58 66 66 90 66 90 f6 c4 02 0f 85 69 03 00 00 31 ff e8 f2 25 90 ff fb 66 66 90 <66> 66 90 45 85 ed 0f 88 37 0 3 00 00 4c 8b 45 d0 4c 2b 45 b8 48 ba [220/488] [ 1432.472009] RSP: 0000:ffffb1d70c527e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [ 1432.481218] RAX: ffff95b3ff862a00 RBX: ffffffff8d1449c0 RCX: 000000000000001f [ 1432.489931] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1432.498633] RBP: ffffb1d70c527e90 R08: ffffd016493f3b42 R09: ffff95b3ff861a64 [ 1432.507337] R10: ffffb1d70c527e18 R11: 0000000000000050 R12: ffffd1aaefa40090 [ 1432.516024] R13: 0000000000000001 R14: 0000000000000001 R15: ffffffff8d1449c0 [ 1432.524693] ? cpuidle_enter_state+0xa1/0x460 [ 1432.530249] cpuidle_enter+0x17/0x20 [ 1432.534925] call_cpuidle+0x23/0x40 [ 1432.539486] do_idle+0x201/0x280 [ 1432.543758] cpu_startup_entry+0x1d/0x20 [ 1432.548811] start_secondary+0x1ab/0x200 [ 1432.553856] secondary_startup_64+0xa4/0xb0 [ 1432.559177] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif input_leds kvm_intel cdc_ether usbnet kvm mii irqbypass sch_f q_codel ipmi_si ib_iser ioatdma ipmi_devintf rdma_cm ipmi_msghandler lpc_ich dca wmi mac_hid iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x _tables btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 multipath linear mgag200 i2c_algo_bit dr m_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper be2net ahci megaraid_sas libahci hid_generic usbhid hid [ 1432.637312] ---[ end trace 826a05dfaf1c2407 ]--- [ 1432.643261] RIP: 0010:perf_prepare_sample+0x8f/0x510 [ 1432.649594] Code: ff ff 41 f6 c4 01 0f 85 22 02 00 00 41 f6 c4 20 74 26 4d 85 e4 0f 88 0c 01 00 00 4c 89 f6 4c 89 ff e8 f5 fe ff ff 49 89 45 70 <48> 8b 00 8d 04 c5 08 00 00 0 0 66 01 43 06 41 f7 c4 00 04 00 00 74 [ 1432.672236] RSP: 0000:ffff95b3ff843a78 EFLAGS: 00010082 [ 1432.678918] RAX: 8d1217eb826cce00 RBX: ffff95b3ff843ad8 RCX: 000000000000001f [ 1432.687752] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1432.696591] RBP: ffff95b3ff843ab0 R08: ffffd016493f3b42 R09: 0000000000000000 [ 1432.705428] R10: ffff95b3ff843a08 R11: 0000000000000000 R12: 80000000000306e5 [ 1432.714268] R13: ffff95b3ff843bc0 R14: ffff95b3ff843b18 R15: ffff95b3f6b65800 [ 1432.723109] FS: 0000000000000000(0000) GS:ffff95b3ff840000(0000) knlGS:0000000000000000 [ 1432.733031] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1432.740334] CR2: 000055bf7f768c90 CR3: 0000001fd220e005 CR4: 00000000000606e0 [ 1432.749210] Kernel panic - not syncing: Fatal exception in interrupt [ 1432.757265] Kernel Offset: 0xac00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 1432.770166] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- [ 1432.779582] ------------[ cut here ]------------ [ 1432.785700] sched: Unexpected reschedule of offline CPU#0! [ 1432.792802] WARNING: CPU: 1 PID: 0 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50 [ 1432.803993] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif input_leds kvm_intel cdc_ether usbnet kvm mii irqbypass sch_f q_codel ipmi_si ib_iser ioatdma ipmi_devintf rdma_cm ipmi_msghandler lpc_ich dca wmi mac_hid iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x _tables btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 multipath linear mgag200 i2c_algo_bit dr m_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper be2net ahci megaraid_sas libahci hid_generic usbhid hid [ 1432.884112] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W 5.0.0-rc3-ravi-pfuzzer+ #1 [ 1432.895062] Hardware name: IBM CPU PLANAR -[8722xyz]-/00FL808, BIOS -[KOE162DUS-2.30]- 08/27/2018 [176/488] [ 1432.911307] RIP: 0010:native_smp_send_reschedule+0x3f/0x50 [ 1432.918548] Code: c0 84 c0 74 17 48 8b 05 4f 64 15 01 be fd 00 00 00 48 8b 40 30 e8 31 c4 ba 00 5d c3 89 fe 48 c7 c7 10 8c cb 8c e8 01 65 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0 f 1f 84 00 00 00 00 00 66 66 66 66 90 [ 1432.941803] RSP: 0000:ffff95b3ff843460 EFLAGS: 00010082 [ 1432.948781] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000006 [ 1432.957893] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff95b3ff8563a0 [ 1432.967013] RBP: ffff95b3ff843460 R08: 0000000000000001 R09: 0000000000cdcdcd [ 1432.976127] R10: 000000000005538e R11: 0000000000000001 R12: ffff95b3f67696c0 [ 1432.985244] R13: ffff95b3ff822a00 R14: ffff95b3ff843518 R15: ffff95b3ff822a00 [ 1432.994364] FS: 0000000000000000(0000) GS:ffff95b3ff840000(0000) knlGS:0000000000000000 [ 1433.004567] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1433.012158] CR2: 000055bf7f768c90 CR3: 0000001fd220e005 CR4: 00000000000606e0 [ 1433.021312] Call Trace: [ 1433.025226] [ 1433.028630] resched_curr+0x6c/0xd0 [ 1433.033660] check_preempt_curr+0x54/0x90 [ 1433.039247] ttwu_do_wakeup+0x1e/0x150 [ 1433.044514] ttwu_do_activate+0x77/0x80 [ 1433.049855] try_to_wake_up+0x1d6/0x4b0 [ 1433.055170] ? mga_dirty_update+0x1ed/0x320 [mgag200] [ 1433.061823] default_wake_function+0x12/0x20 [ 1433.067596] autoremove_wake_function+0x12/0x40 [ 1433.073646] __wake_up_common+0x8c/0x130 [ 1433.078998] __wake_up_common_lock+0x80/0xc0 [ 1433.084721] __wake_up+0x13/0x20 [ 1433.089269] wake_up_klogd_work_func+0x40/0x60 [ 1433.095165] irq_work_run_list+0x55/0x80 [ 1433.100472] ? tick_sched_do_timer+0x60/0x60 [ 1433.106178] irq_work_tick+0x40/0x50 [ 1433.111105] update_process_times+0x42/0x60 [ 1433.116696] tick_sched_handle+0x29/0x60 [ 1433.121987] tick_sched_timer+0x3c/0x80 [ 1433.127170] __hrtimer_run_queues+0x106/0x270 [ 1433.132925] hrtimer_interrupt+0x116/0x240 [ 1433.138372] smp_apic_timer_interrupt+0x6f/0x150 [ 1433.144382] apic_timer_interrupt+0xf/0x20 [ 1433.149802] RIP: 0010:panic+0x25b/0x2a4 [ 1433.154921] Code: 83 c3 64 eb a6 83 3d df 4b 88 01 00 74 05 e8 00 87 02 00 48 c7 c6 e0 01 51 8d 48 c7 c7 08 2a cc 8c e8 40 92 06 00 fb 66 66 90 <66> 66 90 31 db e8 af d7 0d 0 0 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 [ 1433.177603] RSP: 0000:ffff95b3ff8438b8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 [ 1433.186893] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006 [ 1433.195683] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff95b3ff8563a0 [ 1433.204434] RBP: ffff95b3ff843930 R08: 0000000000000001 R09: 0000000000cdcdcd [ 1433.213177] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [132/488] [ 1433.221896] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1433.230589] ? apic_timer_interrupt+0xa/0x20 [ 1433.236060] ? panic+0x257/0x2a4 [ 1433.240357] oops_end+0xe4/0xf0 [ 1433.244540] die+0x42/0x50 [ 1433.248220] do_general_protection+0xcc/0x160 [ 1433.253710] general_protection+0x1e/0x30 [ 1433.258808] RIP: 0010:perf_prepare_sample+0x8f/0x510 [ 1433.264972] Code: ff ff 41 f6 c4 01 0f 85 22 02 00 00 41 f6 c4 20 74 26 4d 85 e4 0f 88 0c 01 00 00 4c 89 f6 4c 89 ff e8 f5 fe ff ff 49 89 45 70 <48> 8b 00 8d 04 c5 08 00 00 0 0 66 01 43 06 41 f7 c4 00 04 00 00 74 [ 1433.287267] RSP: 0000:ffff95b3ff843a78 EFLAGS: 00010082 [ 1433.293772] RAX: 8d1217eb826cce00 RBX: ffff95b3ff843ad8 RCX: 000000000000001f [ 1433.302419] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1433.311056] RBP: ffff95b3ff843ab0 R08: ffffd016493f3b42 R09: 0000000000000000 [ 1433.319685] R10: ffff95b3ff843a08 R11: 0000000000000000 R12: 80000000000306e5 [ 1433.328307] R13: ffff95b3ff843bc0 R14: ffff95b3ff843b18 R15: ffff95b3f6b65800 [ 1433.336927] ? perf_prepare_sample+0x29b/0x510 [ 1433.342545] ? intel_pmu_drain_bts_buffer+0x194/0x230 [ 1433.348849] intel_pmu_drain_bts_buffer+0x160/0x230 [ 1433.354963] ? tick_nohz_irq_exit+0x31/0x40 [ 1433.360301] ? smp_call_function_single_interrupt+0x48/0xe0 [ 1433.367202] ? call_function_single_interrupt+0xf/0x20 [ 1433.373620] ? call_function_single_interrupt+0xa/0x20 [ 1433.380031] ? x86_schedule_events+0x1a0/0x2f0 [ 1433.385662] ? x86_pmu_commit_txn+0xb4/0x100 [ 1433.391096] ? find_busiest_group+0x47/0x5d0 [ 1433.396522] ? perf_event_set_state.part.42+0x12/0x50 [ 1433.402829] ? perf_mux_hrtimer_restart+0x40/0xb0 [ 1433.408757] intel_pmu_disable_event+0xae/0x100 [ 1433.414492] ? intel_pmu_disable_event+0xae/0x100 [ 1433.420424] x86_pmu_stop+0x7a/0xb0 [ 1433.424995] x86_pmu_del+0x57/0x120 [ 1433.429550] event_sched_out.isra.101+0x83/0x180 [ 1433.435381] group_sched_out.part.103+0x57/0xe0 [ 1433.441104] ctx_sched_out+0x188/0x240 [ 1433.445963] ctx_resched+0xa8/0xd0 [ 1433.450430] __perf_event_enable+0x193/0x1e0 [ 1433.455870] event_function+0x8e/0xc0 [ 1433.460625] remote_function+0x41/0x50 [ 1433.465477] flush_smp_call_function_queue+0x68/0x100 [ 1433.471784] generic_smp_call_function_single_interrupt+0x13/0x30 [ 1433.479244] smp_call_function_single_interrupt+0x3e/0xe0 [ 1433.485917] call_function_single_interrupt+0xf/0x20 [ 1433.492094] [88/488] [ 1433.495056] RIP: 0010:cpuidle_enter_state+0xc2/0x460 [ 1433.501231] Code: 66 90 31 ff e8 2f 19 8a ff 80 7d c7 00 74 17 9c 58 66 66 90 66 90 f6 c4 02 0f 85 69 03 00 00 31 ff e8 f2 25 90 ff fb 66 66 90 <66> 66 90 45 85 ed 0f 88 37 0 3 00 00 4c 8b 45 d0 4c 2b 45 b8 48 ba [ 1433.523542] RSP: 0000:ffffb1d70c527e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [ 1433.532691] RAX: ffff95b3ff862a00 RBX: ffffffff8d1449c0 RCX: 000000000000001f [ 1433.541355] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1433.550023] RBP: ffffb1d70c527e90 R08: ffffd016493f3b42 R09: ffff95b3ff861a64 [ 1433.558700] R10: ffffb1d70c527e18 R11: 0000000000000050 R12: ffffd1aaefa40090 [ 1433.567376] R13: 0000000000000001 R14: 0000000000000001 R15: ffffffff8d1449c0 [ 1433.576065] ? cpuidle_enter_state+0xa1/0x460 [ 1433.581651] cpuidle_enter+0x17/0x20 [ 1433.586361] call_cpuidle+0x23/0x40 [ 1433.590943] do_idle+0x201/0x280 [ 1433.595219] cpu_startup_entry+0x1d/0x20 [ 1433.600256] start_secondary+0x1ab/0x200 [ 1433.605284] secondary_startup_64+0xa4/0xb0 [ 1433.610599] ---[ end trace 826a05dfaf1c2408 ]--- [ 1433.616404] ------------[ cut here ]------------ [ 1433.622203] sched: Unexpected reschedule of offline CPU#2! [ 1433.628979] WARNING: CPU: 1 PID: 0 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x3f/0x50 [ 1433.639842] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp ipmi_ssif input_leds kvm_intel cdc_ether usbnet kvm mii irqbypass sch_f q_codel ipmi_si ib_iser ioatdma ipmi_devintf rdma_cm ipmi_msghandler lpc_ich dca wmi mac_hid iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x _tables btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx libcrc32c xor raid6_pq raid1 raid0 multipath linear mgag200 i2c_algo_bit dr m_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper be2net ahci megaraid_sas libahci hid_generic usbhid hid [ 1433.717991] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W 5.0.0-rc3-ravi-pfuzzer+ #1 [ 1433.728625] Hardware name: IBM CPU PLANAR -[8722xyz]-/00FL808, BIOS -[KOE162DUS-2.30]- 08/27/2018 [ 1433.744299] RIP: 0010:native_smp_send_reschedule+0x3f/0x50 [ 1433.751279] Code: c0 84 c0 74 17 48 8b 05 4f 64 15 01 be fd 00 00 00 48 8b 40 30 e8 31 c4 ba 00 5d c3 89 fe 48 c7 c7 10 8c cb 8c e8 01 65 03 00 <0f> 0b 5d c3 0f 1f 00 66 2e 0 f 1f 84 00 00 00 00 00 66 66 66 66 90 [ 1433.774036] RSP: 0000:ffff95b3ff843460 EFLAGS: 00010082 [ 1433.780773] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000006 [ 1433.789657] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff95b3ff8563a0 [ 1433.798540] RBP: ffff95b3ff843460 R08: 0000000000000001 R09: 0000000000cdcdcd [ 1433.807431] R10: 0000000000100000 R11: 0000000000000001 R12: ffff95c3f82b5b00 [ 1433.816326] R13: ffff95b3ff8a2a00 R14: ffff95b3ff843518 R15: ffff95b3ff8a2a00 [ 1433.825224] FS: 0000000000000000(0000) GS:ffff95b3ff840000(0000) knlGS:0000000000000000 [ 1433.835206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1433.842569] CR2: 000055bf7f768c90 CR3: 0000001fd220e005 CR4: 00000000000606e0 [ 1433.851498] Call Trace: [ 1433.855190] [ 1433.858386] resched_curr+0x6c/0xd0 [ 1433.863229] check_preempt_curr+0x54/0x90 [ 1433.868651] ttwu_do_wakeup+0x1e/0x150 [44/488] [ 1433.873772] ttwu_do_activate+0x77/0x80 [ 1433.878978] try_to_wake_up+0x1d6/0x4b0 [ 1433.884189] ? mga_dirty_update+0x1ed/0x320 [mgag200] [ 1433.890742] default_wake_function+0x12/0x20 [ 1433.896400] autoremove_wake_function+0x12/0x40 [ 1433.902336] __wake_up_common+0x8c/0x130 [ 1433.907585] __wake_up_common_lock+0x80/0xc0 [ 1433.913215] __wake_up+0x13/0x20 [ 1433.917668] wake_up_klogd_work_func+0x40/0x60 [ 1433.923468] irq_work_run_list+0x55/0x80 [ 1433.928694] ? tick_sched_do_timer+0x60/0x60 [ 1433.934299] irq_work_tick+0x40/0x50 [ 1433.939125] update_process_times+0x42/0x60 [ 1433.944637] tick_sched_handle+0x29/0x60 [ 1433.949858] tick_sched_timer+0x3c/0x80 [ 1433.954981] __hrtimer_run_queues+0x106/0x270 [ 1433.960687] hrtimer_interrupt+0x116/0x240 [ 1433.966101] smp_apic_timer_interrupt+0x6f/0x150 [ 1433.972099] apic_timer_interrupt+0xf/0x20 [ 1433.977515] RIP: 0010:panic+0x25b/0x2a4 [ 1433.982634] Code: 83 c3 64 eb a6 83 3d df 4b 88 01 00 74 05 e8 00 87 02 00 48 c7 c6 e0 01 51 8d 48 c7 c7 08 2a cc 8c e8 40 92 06 00 fb 66 66 90 <66> 66 90 31 db e8 af d7 0d 0 0 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 [ 1434.005315] RSP: 0000:ffff95b3ff8438b8 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff13 [ 1434.014609] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006 [ 1434.023395] RDX: 0000000000000000 RSI: 0000000000000082 RDI: ffff95b3ff8563a0 [ 1434.032160] RBP: ffff95b3ff843930 R08: 0000000000000001 R09: 0000000000cdcdcd [ 1434.040903] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000 [ 1434.049620] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 1434.058311] ? apic_timer_interrupt+0xa/0x20 [ 1434.063781] ? panic+0x257/0x2a4 [ 1434.068072] oops_end+0xe4/0xf0 [ 1434.072253] die+0x42/0x50 [ 1434.075927] do_general_protection+0xcc/0x160 [ 1434.081426] general_protection+0x1e/0x30 [ 1434.086523] RIP: 0010:perf_prepare_sample+0x8f/0x510 [ 1434.092683] Code: ff ff 41 f6 c4 01 0f 85 22 02 00 00 41 f6 c4 20 74 26 4d 85 e4 0f 88 0c 01 00 00 4c 89 f6 4c 89 ff e8 f5 fe ff ff 49 89 45 70 <48> 8b 00 8d 04 c5 08 00 00 0 0 66 01 43 06 41 f7 c4 00 04 00 00 74 [ 1434.114974] RSP: 0000:ffff95b3ff843a78 EFLAGS: 00010082 [ 1434.121474] RAX: 8d1217eb826cce00 RBX: ffff95b3ff843ad8 RCX: 000000000000001f [ 1434.130119] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1434.138755] RBP: ffff95b3ff843ab0 R08: ffffd016493f3b42 R09: 0000000000000000 [ 1434.147383] R10: ffff95b3ff843a08 R11: 0000000000000000 R12: 80000000000306e5 [ 1434.156003] R13: ffff95b3ff843bc0 R14: ffff95b3ff843b18 R15: ffff95b3f6b65800 [ 1434.164622] ? perf_prepare_sample+0x29b/0x510 [ 1434.170239] ? intel_pmu_drain_bts_buffer+0x194/0x230 [ 1434.176543] intel_pmu_drain_bts_buffer+0x160/0x230 [ 1434.182655] ? tick_nohz_irq_exit+0x31/0x40 [ 1434.187991] ? smp_call_function_single_interrupt+0x48/0xe0 [ 1434.194890] ? call_function_single_interrupt+0xf/0x20 [ 1434.201311] ? call_function_single_interrupt+0xa/0x20 [ 1434.207712] ? x86_schedule_events+0x1a0/0x2f0 [ 1434.213344] ? x86_pmu_commit_txn+0xb4/0x100 [ 1434.218780] ? find_busiest_group+0x47/0x5d0 [ 1434.224212] ? perf_event_set_state.part.42+0x12/0x50 [ 1434.230528] ? perf_mux_hrtimer_restart+0x40/0xb0 [ 1434.236447] intel_pmu_disable_event+0xae/0x100 [ 1434.242181] ? intel_pmu_disable_event+0xae/0x100 [ 1434.248104] x86_pmu_stop+0x7a/0xb0 [ 1434.252674] x86_pmu_del+0x57/0x120 [ 1434.257230] event_sched_out.isra.101+0x83/0x180 [ 1434.263062] group_sched_out.part.103+0x57/0xe0 [ 1434.268793] ctx_sched_out+0x188/0x240 [ 1434.273654] ctx_resched+0xa8/0xd0 [ 1434.278122] __perf_event_enable+0x193/0x1e0 [ 1434.283563] event_function+0x8e/0xc0 [ 1434.288319] remote_function+0x41/0x50 [ 1434.293170] flush_smp_call_function_queue+0x68/0x100 [ 1434.299485] generic_smp_call_function_single_interrupt+0x13/0x30 [ 1434.306951] smp_call_function_single_interrupt+0x3e/0xe0 [ 1434.313626] call_function_single_interrupt+0xf/0x20 [ 1434.319802] [ 1434.322763] RIP: 0010:cpuidle_enter_state+0xc2/0x460 [ 1434.328936] Code: 66 90 31 ff e8 2f 19 8a ff 80 7d c7 00 74 17 9c 58 66 66 90 66 90 f6 c4 02 0f 85 69 03 00 00 31 ff e8 f2 25 90 ff fb 66 66 90 <66> 66 90 45 85 ed 0f 88 37 03 00 00 4c 8b 45 d0 4c 2b 45 b8 48 ba [ 1434.351246] RSP: 0000:ffffb1d70c527e48 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04 [ 1434.360394] RAX: ffff95b3ff862a00 RBX: ffffffff8d1449c0 RCX: 000000000000001f [ 1434.369055] RDX: 0000000000000000 RSI: 00000000355563e5 RDI: 0000000000000000 [ 1434.377721] RBP: ffffb1d70c527e90 R08: ffffd016493f3b42 R09: ffff95b3ff861a64 [ 1434.386398] R10: ffffb1d70c527e18 R11: 0000000000000050 R12: ffffd1aaefa40090 [ 1434.395071] R13: 0000000000000001 R14: 0000000000000001 R15: ffffffff8d1449c0 [ 1434.403760] ? cpuidle_enter_state+0xa1/0x460 [ 1434.409346] cpuidle_enter+0x17/0x20 [ 1434.414054] call_cpuidle+0x23/0x40 [ 1434.418644] do_idle+0x201/0x280 [ 1434.422918] cpu_startup_entry+0x1d/0x20 [ 1434.427955] start_secondary+0x1ab/0x200 [ 1434.432983] secondary_startup_64+0xa4/0xb0 [ 1434.438301] ---[ end trace 826a05dfaf1c2409 ]--- Thanks, Ravi