From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S964906Ab3BTVCT (ORCPT ); Wed, 20 Feb 2013 16:02:19 -0500 Received: from mail.skyhub.de ([78.46.96.112]:58927 "EHLO mail.skyhub.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S935243Ab3BTVCQ (ORCPT ); Wed, 20 Feb 2013 16:02:16 -0500 Date: Wed, 20 Feb 2013 22:02:10 +0100 From: Borislav Petkov To: Frederic Weisbecker Cc: lkml Subject: nohz splat Message-ID: <20130220210210.GA1395@pd.tnic> Mail-Followup-To: Borislav Petkov , Frederic Weisbecker , lkml MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi, I just gave your 3.8-rc6-nohz4 branch a run, here's what I got: > [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8.0-rc6+ root=/dev/sda7 ro rcu_nocbs=4-7 full_nohz=4-7 log_bug_len=10M resume=/dev/sdb5 no_console_suspend ignore_loglevel Right, so I'm using cores 4-7 for nohz. > [ 0.000000] e820: BIOS-provided physical RAM map: > [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000099bff] usable > [ 0.000000] BIOS-e820: [mem 0x0000000000099c00-0x000000000009ffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved > [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000001febdfff] usable > [ 0.000000] BIOS-e820: [mem 0x000000001febe000-0x000000001fffffff] ACPI NVS > [ 0.000000] BIOS-e820: [mem 0x0000000020000000-0x000000002fffcfff] usable > [ 0.000000] BIOS-e820: [mem 0x000000002fffd000-0x000000002fffffff] ACPI data > [ 0.000000] BIOS-e820: [mem 0x0000000030000000-0x00000000aa77bfff] usable > [ 0.000000] BIOS-e820: [mem 0x00000000aa77c000-0x00000000ac813fff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000ac814000-0x00000000ad7fffff] usable > [ 0.000000] BIOS-e820: [mem 0x00000000b0000000-0x00000000b3ffffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000fed20000-0x00000000fed3ffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000fed50000-0x00000000fed8ffff] reserved > [ 0.000000] BIOS-e820: [mem 0x00000000ffa00000-0x00000000ffa3ffff] reserved > [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000044fffffff] usable ... > [ 0.065958] SMP alternatives: lockdep: fixing up alternatives > [ 0.079447] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter. > [ 0.066024] smpboot: Booting Node 0, Processors #1 > [ 0.079767] SMP alternatives: lockdep: fixing up alternatives > [ 0.079875] #2 > [ 0.093295] SMP alternatives: lockdep: fixing up alternatives > [ 0.093405] #3 > [ 0.106832] SMP alternatives: lockdep: fixing up alternatives > [ 0.118047] ------------[ cut here ]------------ > [ 0.118055] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x58/0x60() This is WARN_ON(1) in native_smp_send_reschedule() which screams because those cores are somehow offline. > [ 0.118056] Hardware name: Precision T3600 > [ 0.106943] #4 > [ 0.118056] Modules linked in: > [ 0.118059] Pid: 0, comm: swapper/4 Not tainted 3.8.0-rc6+ #4 > [ 0.118060] Call Trace: > [ 0.118062] [] warn_slowpath_common+0x7f/0xc0 > [ 0.118065] [] warn_slowpath_null+0x1a/0x20 > [ 0.118067] [] native_smp_send_reschedule+0x58/0x60 > [ 0.118069] [] wake_up_nohz_cpu+0x80/0x90 > [ 0.118073] [] add_timer_on+0x91/0x110 > [ 0.118076] [] mce_start_timer.isra.12+0x6b/0x80 > [ 0.118078] [] __mcheck_cpu_init_timer+0x5d/0x70 > [ 0.118080] [] mcheck_cpu_init+0x36b/0x400 > [ 0.118083] [] identify_cpu+0x39d/0x3d0 > [ 0.118085] [] identify_secondary_cpu+0x14/0x1b > [ 0.118087] [] smp_store_cpu_info+0x38/0x3a > [ 0.118089] [] start_secondary+0xfb/0x1f7 > [ 0.118107] ---[ end trace b3465549d7aae543 ]--- > [ 0.120474] SMP alternatives: lockdep: fixing up alternatives > [ 0.133276] ------------[ cut here ]------------ > [ 0.133284] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x58/0x60() > [ 0.133284] Hardware name: Precision T3600 > [ 0.122175] #5 > [ 0.133285] Modules linked in: > [ 0.133287] Pid: 0, comm: swapper/5 Tainted: G W 3.8.0-rc6+ #4 > [ 0.133288] Call Trace: > [ 0.133290] [] warn_slowpath_common+0x7f/0xc0 > [ 0.133293] [] warn_slowpath_null+0x1a/0x20 > [ 0.133294] [] native_smp_send_reschedule+0x58/0x60 > [ 0.133296] [] wake_up_nohz_cpu+0x80/0x90 > [ 0.133299] [] add_timer_on+0x91/0x110 > [ 0.133301] [] mce_start_timer.isra.12+0x6b/0x80 > [ 0.133303] [] __mcheck_cpu_init_timer+0x5d/0x70 > [ 0.133305] [] mcheck_cpu_init+0x36b/0x400 > [ 0.133307] [] identify_cpu+0x39d/0x3d0 > [ 0.133309] [] identify_secondary_cpu+0x14/0x1b > [ 0.133310] [] smp_store_cpu_info+0x38/0x3a > [ 0.133312] [] start_secondary+0xfb/0x1f7 > [ 0.133314] ---[ end trace b3465549d7aae544 ]--- > [ 0.135578] SMP alternatives: lockdep: fixing up alternatives > [ 0.148375] ------------[ cut here ]------------ > [ 0.148383] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x58/0x60() > [ 0.148383] Hardware name: Precision T3600 > [ 0.137282] #6 > [ 0.148384] Modules linked in: > [ 0.148386] Pid: 0, comm: swapper/6 Tainted: G W 3.8.0-rc6+ #4 > [ 0.148387] Call Trace: > [ 0.148389] [] warn_slowpath_common+0x7f/0xc0 > [ 0.148392] [] warn_slowpath_null+0x1a/0x20 > [ 0.148393] [] native_smp_send_reschedule+0x58/0x60 > [ 0.148395] [] wake_up_nohz_cpu+0x80/0x90 > [ 0.148398] [] add_timer_on+0x91/0x110 > [ 0.148401] [] mce_start_timer.isra.12+0x6b/0x80 > [ 0.148403] [] __mcheck_cpu_init_timer+0x5d/0x70 > [ 0.148404] [] mcheck_cpu_init+0x36b/0x400 > [ 0.148407] [] identify_cpu+0x39d/0x3d0 > [ 0.148408] [] identify_secondary_cpu+0x14/0x1b > [ 0.148410] [] smp_store_cpu_info+0x38/0x3a > [ 0.148411] [] start_secondary+0xfb/0x1f7 > [ 0.148414] ---[ end trace b3465549d7aae545 ]--- > [ 0.150691] SMP alternatives: lockdep: fixing up alternatives > [ 0.152382] #7 OK > [ 0.163490] ------------[ cut here ]------------ > [ 0.163498] WARNING: at arch/x86/kernel/smp.c:123 native_smp_send_reschedule+0x58/0x60() > [ 0.163499] Hardware name: Precision T3600 > [ 0.163500] Modules linked in: > [ 0.163501] Pid: 0, comm: swapper/7 Tainted: G W 3.8.0-rc6+ #4 > [ 0.163502] Call Trace: > [ 0.163506] [] warn_slowpath_common+0x7f/0xc0 > [ 0.163508] [] warn_slowpath_null+0x1a/0x20 > [ 0.163509] [] native_smp_send_reschedule+0x58/0x60 > [ 0.163512] [] wake_up_nohz_cpu+0x80/0x90 > [ 0.163515] [] add_timer_on+0x91/0x110 > [ 0.163517] [] mce_start_timer.isra.12+0x6b/0x80 > [ 0.163518] [] __mcheck_cpu_init_timer+0x5d/0x70 > [ 0.163521] [] mcheck_cpu_init+0x36b/0x400 > [ 0.163523] [] identify_cpu+0x39d/0x3d0 > [ 0.163524] [] identify_secondary_cpu+0x14/0x1b > [ 0.163526] [] smp_store_cpu_info+0x38/0x3a > [ 0.163527] [] start_secondary+0xfb/0x1f7 > [ 0.163528] ---[ end trace b3465549d7aae546 ]--- > [ 0.165587] Brought up 8 CPUs Right, those splats happen on all 4-7 cores which we marked for nohz with the boot cmdline. ... > [ 8.170292] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready > [ 9.731396] e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: Rx/Tx > [ 9.731424] e1000e 0000:00:19.0 eth0: 10/100 speed: disabling TSO > [ 9.731478] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > [ 240.323418] INFO: task rcuo4:10 blocked for more than 120 seconds. and here's where the rcu offloading threads hang too. Also on those cores 4-7. > [ 240.323450] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 240.323466] rcuo4 D 0000000000000000 0 10 2 0x00000000 > [ 240.323492] ffff88043d563d98 0000000000000006 ffff88043d563fd8 ffff88043d558f80 > [ 240.323521] ffff88043d563fd8 ffff88043d563fd8 ffff88043d559f00 ffff88043d558f80 > [ 240.323549] ffff88043d563da8 ffff88043d4cddd0 ffff88044fd0df40 ffff88044fd0df40 > [ 240.323577] Call Trace: > [ 240.323595] [] schedule+0x29/0x70 > [ 240.323612] [] rcu_nocb_kthread+0x1af/0x200 > [ 240.323629] [] ? __init_waitqueue_head+0x60/0x60 > [ 240.323646] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 240.323662] [] kthread+0xea/0xf0 > [ 240.323677] [] ? kthread_create_on_node+0x160/0x160 > [ 240.323694] [] ret_from_fork+0x7c/0xb0 > [ 240.323708] [] ? kthread_create_on_node+0x160/0x160 > [ 240.323721] INFO: lockdep is turned off. > [ 240.323733] INFO: task rcuo5:11 blocked for more than 120 seconds. > [ 240.323745] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 240.323759] rcuo5 D 0000000000000000 0 11 2 0x00000000 > [ 240.323781] ffff88043d565d98 0000000000000006 ffff88043d565fd8 ffff88043d559f00 > [ 240.323811] ffff88043d565fd8 ffff88043d565fd8 ffff88043d55ae80 ffff88043d559f00 > [ 240.323839] ffff88043d565da8 ffff88043d4cddd0 ffff88044fd4df40 ffff88044fd4df40 > [ 240.323867] Call Trace: > [ 240.323878] [] schedule+0x29/0x70 > [ 240.323893] [] rcu_nocb_kthread+0x1af/0x200 > [ 240.323908] [] ? __init_waitqueue_head+0x60/0x60 > [ 240.323924] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 240.323939] [] kthread+0xea/0xf0 > [ 240.323953] [] ? kthread_create_on_node+0x160/0x160 > [ 240.323969] [] ret_from_fork+0x7c/0xb0 > [ 240.323983] [] ? kthread_create_on_node+0x160/0x160 > [ 240.323996] INFO: lockdep is turned off. > [ 240.324007] INFO: task rcuo6:12 blocked for more than 120 seconds. > [ 240.324019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 240.324032] rcuo6 D 0000000000000000 0 12 2 0x00000000 > [ 240.324053] ffff88043d567d98 0000000000000006 ffff88043d567fd8 ffff88043d55ae80 > [ 240.324081] ffff88043d567fd8 ffff88043d567fd8 ffff88043d55be00 ffff88043d55ae80 > [ 240.324109] ffff88043d567da8 ffff88043d4cddd0 ffff88044fd8df40 ffff88044fd8df40 > [ 240.324137] Call Trace: > [ 240.324148] [] schedule+0x29/0x70 > [ 240.324162] [] rcu_nocb_kthread+0x1af/0x200 > [ 240.324177] [] ? __init_waitqueue_head+0x60/0x60 > [ 240.324193] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 240.324208] [] kthread+0xea/0xf0 > [ 240.324222] [] ? kthread_create_on_node+0x160/0x160 > [ 240.324238] [] ret_from_fork+0x7c/0xb0 > [ 240.324252] [] ? kthread_create_on_node+0x160/0x160 > [ 240.324265] INFO: lockdep is turned off. > [ 240.324275] INFO: task rcuo7:13 blocked for more than 120 seconds. > [ 240.324287] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 240.324301] rcuo7 D 0000000000000000 0 13 2 0x00000000 > [ 240.324322] ffff88043d569d98 0000000000000006 ffff88043d569fd8 ffff88043d55be00 > [ 240.324350] ffff88043d569fd8 ffff88043d569fd8 ffff88043d55cd80 ffff88043d55be00 > [ 240.324377] ffff88043d569da8 ffff88043d4cddd0 ffff88044fdcdf40 ffff88044fdcdf40 > [ 240.324405] Call Trace: > [ 240.324416] [] schedule+0x29/0x70 > [ 240.324430] [] rcu_nocb_kthread+0x1af/0x200 > [ 240.324445] [] ? __init_waitqueue_head+0x60/0x60 > [ 240.324461] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 240.324477] [] kthread+0xea/0xf0 > [ 240.324494] [] ? kthread_create_on_node+0x160/0x160 > [ 240.325129] [] ret_from_fork+0x7c/0xb0 > [ 240.325760] [] ? kthread_create_on_node+0x160/0x160 > [ 240.326388] INFO: lockdep is turned off. > [ 360.268728] INFO: task rcuo4:10 blocked for more than 120 seconds. > [ 360.269381] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 360.270010] rcuo4 D 0000000000000000 0 10 2 0x00000000 > [ 360.270642] ffff88043d563d98 0000000000000006 ffff88043d563fd8 ffff88043d558f80 > [ 360.271273] ffff88043d563fd8 ffff88043d563fd8 ffff88043d559f00 ffff88043d558f80 > [ 360.271901] ffff88043d563da8 ffff88043d4cddd0 ffff88044fd0df40 ffff88044fd0df40 > [ 360.272532] Call Trace: > [ 360.273196] [] schedule+0x29/0x70 > [ 360.273822] [] rcu_nocb_kthread+0x1af/0x200 > [ 360.274441] [] ? __init_waitqueue_head+0x60/0x60 > [ 360.275056] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 360.275667] [] kthread+0xea/0xf0 > [ 360.276270] [] ? kthread_create_on_node+0x160/0x160 > [ 360.276884] [] ret_from_fork+0x7c/0xb0 > [ 360.277491] [] ? kthread_create_on_node+0x160/0x160 > [ 360.278097] INFO: lockdep is turned off. > [ 360.278709] INFO: task rcuo5:11 blocked for more than 120 seconds. > [ 360.279321] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 360.279939] rcuo5 D 0000000000000000 0 11 2 0x00000000 > [ 360.280566] ffff88043d565d98 0000000000000006 ffff88043d565fd8 ffff88043d559f00 > [ 360.281216] ffff88043d565fd8 ffff88043d565fd8 ffff88043d55ae80 ffff88043d559f00 > [ 360.281859] ffff88043d565da8 ffff88043d4cddd0 ffff88044fd4df40 ffff88044fd4df40 > [ 360.282513] Call Trace: > [ 360.283158] [] schedule+0x29/0x70 > [ 360.283808] [] rcu_nocb_kthread+0x1af/0x200 > [ 360.284464] [] ? __init_waitqueue_head+0x60/0x60 > [ 360.285133] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 360.285780] [] kthread+0xea/0xf0 > [ 360.286434] [] ? kthread_create_on_node+0x160/0x160 > [ 360.287084] [] ret_from_fork+0x7c/0xb0 > [ 360.287739] [] ? kthread_create_on_node+0x160/0x160 > [ 360.288397] INFO: lockdep is turned off. > [ 360.289066] INFO: task rcuo6:12 blocked for more than 120 seconds. > [ 360.289713] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 360.290382] rcuo6 D 0000000000000000 0 12 2 0x00000000 > [ 360.291050] ffff88043d567d98 0000000000000006 ffff88043d567fd8 ffff88043d55ae80 > [ 360.291727] ffff88043d567fd8 ffff88043d567fd8 ffff88043d55be00 ffff88043d55ae80 > [ 360.292394] ffff88043d567da8 ffff88043d4cddd0 ffff88044fd8df40 ffff88044fd8df40 > [ 360.293071] Call Trace: > [ 360.293708] [] schedule+0x29/0x70 > [ 360.294360] [] rcu_nocb_kthread+0x1af/0x200 > [ 360.295021] [] ? __init_waitqueue_head+0x60/0x60 > [ 360.295682] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 360.296343] [] kthread+0xea/0xf0 And from here on now, all goes to hell..., machine is still responsive though. > [ 360.297007] ------------[ cut here ]------------ > [ 360.299716] WARNING: at kernel/sched/sched.h:521 sched_avg_update+0x116/0x120() > [ 360.302453] Hardware name: Precision T3600 > [ 360.305186] Modules linked in: loop snd_hda_codec_hdmi usbhid coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel snd_hda_codec_realtek xts aes_x86_64 lrw snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd gf128mul ehci_pci ablk_helper cryptd ehci_hcd xhci_hcd e1000e sb_edac edac_core iTCO_wdt usbcore acpi_cpufreq microcode mperf iTCO_vendor_support usb_common dcdbas lpc_ich soundcore button pcspkr evdev i2c_i801 mfd_core processor > [ 360.311579] Pid: 64, comm: khungtaskd Tainted: G W 3.8.0-rc6+ #4 > [ 360.314686] Call Trace: > [ 360.317767] [] warn_slowpath_common+0x7f/0xc0 > [ 360.320895] [] warn_slowpath_null+0x1a/0x20 > [ 360.323988] [] sched_avg_update+0x116/0x120 > [ 360.327066] [] __update_cpu_load+0xc8/0x120 > [ 360.330153] [] scheduler_tick+0x7a/0x150 > [ 360.333240] [] update_process_times+0x67/0x80 > [ 360.336311] [] tick_sched_handle.isra.12+0x3c/0x70 > [ 360.339373] [] tick_sched_timer+0x4c/0x80 > [ 360.342434] [] __run_hrtimer+0x86/0x230 > [ 360.345481] [] ? tick_sched_do_timer+0x80/0x80 > [ 360.348527] [] hrtimer_interrupt+0xf7/0x230 > [ 360.351580] [] ? _raw_spin_unlock+0x2b/0x50 > [ 360.354631] [] smp_apic_timer_interrupt+0x69/0x99 > [ 360.357685] [] apic_timer_interrupt+0x6f/0x80 > [ 360.360725] [] ? vprintk_emit+0x1de/0x580 > [ 360.363790] [] ? trace_hardirqs_on+0xd/0x10 > [ 360.366830] [] ? vprintk_emit+0x1e2/0x580 > [ 360.369837] [] ? kthread+0xea/0xf0 > [ 360.372812] [] printk+0x4d/0x4f > [ 360.375770] [] ? kthread+0xea/0xf0 > [ 360.378722] [] ? kthread+0xea/0xf0 > [ 360.381656] [] printk_address+0x31/0x40 > [ 360.384564] [] print_trace_address+0x3c/0x50 > [ 360.387465] [] ? kthread+0xea/0xf0 > [ 360.390367] [] print_context_stack+0xdf/0xf0 > [ 360.393276] [] dump_trace+0x1af/0x2d0 > [ 360.396164] [] show_trace_log_lvl+0x5c/0x80 > [ 360.399055] [] show_stack_log_lvl+0xbf/0x1b0 > [ 360.401924] [] show_stack+0x1c/0x20 > [ 360.404784] [] sched_show_task+0xfd/0x170 > [ 360.407632] [] ? sched_show_task+0x6a/0x170 > [ 360.410464] [] ? vprintk_emit+0x1de/0x580 > [ 360.413295] [] watchdog+0x2d0/0x340 > [ 360.416082] [] ? watchdog+0x86/0x340 > [ 360.418837] [] ? __schedule+0x2fb/0x7d0 > [ 360.421581] [] ? hung_task_panic+0x20/0x20 > [ 360.424326] [] kthread+0xea/0xf0 > [ 360.427073] [] ? kthread_create_on_node+0x160/0x160 > [ 360.429813] [] ret_from_fork+0x7c/0xb0 > [ 360.432544] [] ? kthread_create_on_node+0x160/0x160 > [ 360.435292] ---[ end trace b3465549d7aae548 ]--- > [ 360.438035] ------------[ cut here ]------------ > [ 360.440771] WARNING: at kernel/sched/sched.h:521 update_curr+0x156/0x1e0() > [ 360.443538] Hardware name: Precision T3600 > [ 360.446294] Modules linked in: loop snd_hda_codec_hdmi usbhid coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel snd_hda_codec_realtek xts aes_x86_64 lrw snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd gf128mul ehci_pci ablk_helper cryptd ehci_hcd xhci_hcd e1000e sb_edac edac_core iTCO_wdt usbcore acpi_cpufreq microcode mperf iTCO_vendor_support usb_common dcdbas lpc_ich soundcore button pcspkr evdev i2c_i801 mfd_core processor > [ 360.452717] Pid: 64, comm: khungtaskd Tainted: G W 3.8.0-rc6+ #4 > [ 360.455854] Call Trace: > [ 360.458957] [] warn_slowpath_common+0x7f/0xc0 > [ 360.462110] [] warn_slowpath_null+0x1a/0x20 > [ 360.465231] [] update_curr+0x156/0x1e0 > [ 360.468339] [] task_tick_fair+0x52/0x130 > [ 360.471451] [] scheduler_tick+0xd6/0x150 > [ 360.474559] [] update_process_times+0x67/0x80 > [ 360.477649] [] tick_sched_handle.isra.12+0x3c/0x70 > [ 360.480738] [] tick_sched_timer+0x4c/0x80 > [ 360.483819] [] __run_hrtimer+0x86/0x230 > [ 360.486897] [] ? tick_sched_do_timer+0x80/0x80 > [ 360.489976] [] hrtimer_interrupt+0xf7/0x230 > [ 360.493051] [] ? _raw_spin_unlock+0x2b/0x50 > [ 360.496125] [] smp_apic_timer_interrupt+0x69/0x99 > [ 360.499181] [] apic_timer_interrupt+0x6f/0x80 > [ 360.502218] [] ? vprintk_emit+0x1de/0x580 > [ 360.505158] [] ? trace_hardirqs_on+0xd/0x10 > [ 360.507965] [] ? vprintk_emit+0x1e2/0x580 > [ 360.510638] [] ? kthread+0xea/0xf0 > [ 360.513283] [] printk+0x4d/0x4f > [ 360.515908] [] ? kthread+0xea/0xf0 > [ 360.518517] [] ? kthread+0xea/0xf0 > [ 360.521100] [] printk_address+0x31/0x40 > [ 360.523689] [] print_trace_address+0x3c/0x50 > [ 360.526273] [] ? kthread+0xea/0xf0 > [ 360.528850] [] print_context_stack+0xdf/0xf0 > [ 360.531433] [] dump_trace+0x1af/0x2d0 > [ 360.534010] [] show_trace_log_lvl+0x5c/0x80 > [ 360.536593] [] show_stack_log_lvl+0xbf/0x1b0 > [ 360.539156] [] show_stack+0x1c/0x20 > [ 360.541702] [] sched_show_task+0xfd/0x170 > [ 360.544254] [] ? sched_show_task+0x6a/0x170 > [ 360.546797] [] ? vprintk_emit+0x1de/0x580 > [ 360.549333] [] watchdog+0x2d0/0x340 > [ 360.551851] [] ? watchdog+0x86/0x340 > [ 360.554345] [] ? __schedule+0x2fb/0x7d0 > [ 360.556841] [] ? hung_task_panic+0x20/0x20 > [ 360.559344] [] kthread+0xea/0xf0 > [ 360.561855] [] ? kthread_create_on_node+0x160/0x160 > [ 360.564382] [] ret_from_fork+0x7c/0xb0 > [ 360.566930] [] ? kthread_create_on_node+0x160/0x160 > [ 360.569497] ---[ end trace b3465549d7aae549 ]--- > [ 360.572090] [] ? kthread_create_on_node+0x160/0x160 > [ 360.572332] [] ret_from_fork+0x7c/0xb0 > [ 360.572686] [] ? kthread_create_on_node+0x160/0x160 > [ 360.573334] INFO: lockdep is turned off. > [ 360.573993] INFO: task rcuo7:13 blocked for more than 120 seconds. > [ 360.574647] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 360.575322] rcuo7 D 0000000000000000 0 13 2 0x00000000 > [ 360.576011] ffff88043d569d98 0000000000000006 ffff88043d569fd8 ffff88043d55be00 > [ 360.576716] ffff88043d569fd8 ffff88043d569fd8 ffff88043d55cd80 ffff88043d55be00 > [ 360.577400] ffff88043d569da8 ffff88043d4cddd0 ffff88044fdcdf40 ffff88044fdcdf40 > [ 360.578088] Call Trace: > [ 360.578766] [] schedule+0x29/0x70 > [ 360.579446] [] rcu_nocb_kthread+0x1af/0x200 > [ 360.580138] [] ? __init_waitqueue_head+0x60/0x60 > [ 360.580846] [] ? rcu_implicit_dynticks_qs+0xb0/0xb0 > [ 360.581531] [] kthread+0xea/0xf0 > [ 360.582224] [] ? kthread_create_on_node+0x160/0x160 > [ 360.582909] [] ret_from_fork+0x7c/0xb0 > [ 360.583597] [] ? kthread_create_on_node+0x160/0x160 > [ 360.584290] INFO: lockdep is turned off. > [ 360.585038] ------------[ cut here ]------------ > [ 360.585695] WARNING: at kernel/sched/sched.h:521 pick_next_task_stop+0xf3/0x100() > [ 360.586338] Hardware name: Precision T3600 > [ 360.586957] Modules linked in: loop snd_hda_codec_hdmi usbhid coretemp kvm_intel kvm crc32c_intel ghash_clmulni_intel aesni_intel snd_hda_codec_realtek xts aes_x86_64 lrw snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_page_alloc snd_timer snd gf128mul ehci_pci ablk_helper cryptd ehci_hcd xhci_hcd e1000e sb_edac edac_core iTCO_wdt usbcore acpi_cpufreq microcode mperf iTCO_vendor_support usb_common dcdbas lpc_ich soundcore button pcspkr evdev i2c_i801 mfd_core processor > [ 360.588372] Pid: 64, comm: khungtaskd Tainted: G W 3.8.0-rc6+ #4 > [ 360.588992] Call Trace: > [ 360.589686] [] warn_slowpath_common+0x7f/0xc0 > [ 360.590373] [] warn_slowpath_null+0x1a/0x20 > [ 360.591064] [] pick_next_task_stop+0xf3/0x100 > [ 360.591753] [] __schedule+0x7b0/0x7d0 > [ 360.592443] [] schedule+0x29/0x70 > [ 360.593135] [] schedule_timeout+0x14a/0x290 > [ 360.593823] [] ? call_timer_fn+0x1b0/0x1b0 > [ 360.594510] [] schedule_timeout_interruptible+0x1e/0x20 > [ 360.595201] [] watchdog+0x62/0x340 > [ 360.595888] [] ? watchdog+0x86/0x340 > [ 360.596578] [] ? __schedule+0x2fb/0x7d0 > [ 360.597269] [] ? hung_task_panic+0x20/0x20 > [ 360.597954] [] kthread+0xea/0xf0 > [ 360.598640] [] ? kthread_create_on_node+0x160/0x160 > [ 360.599323] [] ret_from_fork+0x7c/0xb0 > [ 360.599999] [] ? kthread_create_on_node+0x160/0x160 > [ 360.600684] ---[ end trace b3465549d7aae54a ]--- HTH. -- Regards/Gruss, Boris. Sent from a fat crate under my desk. Formatting is fine. --