Hi, I've been getting these NETDEV WATCHDOG errors at infrequent intervals (typically every few weeks; sometimes far more often, as has happened this week), but usually at inconvenient times -- making a TV recording over NFS, for example. It's usually at some point when the network and disk subsystem are stressed together. The network doesn't come back, even if I remove and reinsert the network card driver. I even bought a new network card with a different chip, and saw a precisely similar problem from that (I've included that trace below, too). These have happened on a range of kernels since at least 2.6.31. I include a selection of kernel logs, including one from the other network card. I've seen it on distribution kernels (Debian), kernel.org kernels, and the btrfs dev kernel (that's the 2.6.34-81023-g2ebc346 version). Can anyone help me diagnose and fix the problem? Hugo. Oct 30 19:57:09 vlad kernel: ------------[ cut here ]------------ Oct 30 19:57:09 vlad kernel: WARNING: at net/sched/sch_generic.c:258 dev_watchdog+0x104/0x188() Oct 30 19:57:09 vlad kernel: Hardware name: System Product Name Oct 30 19:57:09 vlad kernel: NETDEV WATCHDOG: eth2 (via-rhine): transmit queue 0 timed out Oct 30 19:57:09 vlad kernel: Modules linked in: loop cpufreq_userspace nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 xfs exportfs btrfs zlib_deflate hwmon_vid powernow_k8 mperf sr_mod sbp2 ieee1394 ide_generic ide_gd_mod ide_cd_mod cdrom pl2303 usbserial k8temp i2c_viapro hwmon 8250_pnp evdev i2c_core 8250 serial_core pcspkr usbhid processor button dm_mirror dm_region_hash dm_log dm_snapshot dm_mod usb_storage usb_libusual raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 md_mod ehci_hcd via_rhine uhci_hcd mii usbcore via82cxxx sd_mod ide_pci_generic ide_core floppy thermal fan unix [last unloaded: scsi_wait_scan] Oct 30 19:57:09 vlad kernel: Pid: 10565, comm: btrfs Not tainted 2.6.36 #1 Oct 30 19:57:09 vlad kernel: Call Trace: Oct 30 19:57:09 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 30 19:57:09 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 30 19:57:09 vlad kernel: [] ? warn_slowpath_common+0x78/0x8d Oct 30 19:57:09 vlad kernel: [] ? dev_watchdog+0x0/0x188 Oct 30 19:57:09 vlad kernel: [] ? warn_slowpath_fmt+0x56/0x5e Oct 30 19:57:09 vlad kernel: [] ? __blk_run_queue+0x45/0x72 Oct 30 19:57:09 vlad kernel: [] ? scsi_run_queue+0x1fc/0x254 Oct 30 19:57:09 vlad kernel: [] ? netdev_drivername+0x3b/0x40 Oct 30 19:57:09 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 30 19:57:09 vlad kernel: [] ? print_cpus_offline+0x13/0xdd Oct 30 19:57:09 vlad kernel: [] ? run_timer_softirq+0x14e/0x1c8 Oct 30 19:57:09 vlad kernel: [] ? __do_softirq+0x6b/0xe4 Oct 30 19:57:09 vlad kernel: [] ? tick_dev_program_event+0x2d/0xe9 Oct 30 19:57:09 vlad kernel: [] ? call_softirq+0x1a/0x24 Oct 30 19:57:09 vlad kernel: [] ? do_softirq+0x2f/0x61 Oct 30 19:57:09 vlad kernel: [] ? irq_exit+0x34/0x71 Oct 30 19:57:09 vlad kernel: [] ? smp_apic_timer_interrupt+0x71/0x7b Oct 30 19:57:09 vlad kernel: [] ? apic_timer_interrupt+0x13/0x20 Oct 30 19:57:09 vlad kernel: [] ? cfq_queue_empty+0x0/0x16 Oct 30 19:57:09 vlad kernel: [] ? generic_make_request+0x24d/0x290 Oct 30 19:57:09 vlad kernel: [] ? generic_make_request+0x22f/0x290 Oct 30 19:57:09 vlad kernel: [] ? __btrfs_map_block+0x3fa/0x40b [btrfs] Oct 30 19:57:09 vlad kernel: [] ? submit_bio+0x9d/0xa4 Oct 30 19:57:09 vlad kernel: [] ? btrfs_map_bio+0x177/0x1cf [btrfs] Oct 30 19:57:09 vlad kernel: [] ? submit_one_bio+0x6a/0x92 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? submit_extent_page+0x11d/0x205 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? end_bio_extent_readpage+0x0/0x1d5 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? __extent_read_full_page+0x538/0x5a4 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? end_bio_extent_readpage+0x0/0x1d5 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? __alloc_pages_nodemask+0x2a0/0x5c0 Oct 30 19:57:09 vlad kernel: [] ? btrfs_get_extent+0x0/0x753 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_get_extent+0x0/0x753 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? extent_readpages+0x9f/0xe1 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? __do_page_cache_readahead+0x121/0x1b9 Oct 30 19:57:09 vlad kernel: [] ? ra_submit+0x1c/0x20 Oct 30 19:57:09 vlad kernel: [] ? relocate_file_extent_cluster+0x143/0x321 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? relocate_block_group+0x344/0x4c3 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_clean_old_snapshots+0x7b/0x133 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_relocate_block_group+0xf1/0x21e [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_relocate_chunk+0x60/0x450 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_dev_extent_chunk_offset+0xb6/0xc2 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_shrink_device+0x1ff/0x32c [btrfs] Oct 30 19:57:09 vlad kernel: [] ? dequeue_task_fair+0x2c/0x195 Oct 30 19:57:09 vlad kernel: [] ? btrfs_ioctl_resize+0x1fb/0x230 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? btrfs_ioctl+0x5c0/0x873 [btrfs] Oct 30 19:57:09 vlad kernel: [] ? ptrace_stop+0x100/0x105 Oct 30 19:57:09 vlad kernel: [] ? ptrace_notify+0x68/0x72 Oct 30 19:57:09 vlad kernel: [] ? do_vfs_ioctl+0x41b/0x45c Oct 30 19:57:09 vlad kernel: [] ? sys_ioctl+0x3d/0x5c Oct 30 19:57:09 vlad kernel: [] ? tracesys+0xcf/0xd4 Oct 30 19:57:09 vlad kernel: ---[ end trace 9befef6a80bf1095 ]--- Oct 30 19:57:09 vlad kernel: eth2: Transmit timed out, status 0003, PHY status 786d, resetting... Oct 30 19:57:09 vlad kernel: eth2: link up, 100Mbps, full-duplex, lpa 0xCDE1 Oct 30 19:58:07 vlad kernel: eth2: Transmit timed out, status 0003, PHY status 786d, resetting... Oct 30 19:58:07 vlad kernel: eth2: link up, 100Mbps, full-duplex, lpa 0xCDE1 Oct 24 19:34:41 vlad kernel: ------------[ cut here ]------------ Oct 24 19:34:41 vlad kernel: WARNING: at net/sched/sch_generic.c:258 dev_watchdog+0x104/0x188() Oct 24 19:34:41 vlad kernel: Hardware name: System Product Name Oct 24 19:34:41 vlad kernel: NETDEV WATCHDOG: eth3 (r8169): transmit queue 0 timed out Oct 24 19:34:41 vlad kernel: Modules linked in: cpufreq_userspace nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 xfs exportfs btrfs zlib_deflate hwmon_vid powernow_k8 mperf sr_mod sbp2 ieee1394 ide_generic ide_gd_mod ide_cd_mod cdrom evdev pcspkr k8temp hwmon i2c_viapro pl2303 i2c_core usbserial usbhid 8250_pnp 8250 serial_core processor button dm_mirror dm_region_hash dm_log dm_snapshot dm_mod raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 md_mod usb_storage usb_libusual r8169 ehci_hcd via_rhine uhci_hcd mii usbcore via82cxxx ide_pci_generic sd_mod ide_core floppy thermal fan unix [last unloaded: scsi_wait_scan] Oct 24 19:34:41 vlad kernel: Pid: 0, comm: swapper Not tainted 2.6.36 #1 Oct 24 19:34:41 vlad kernel: Call Trace: Oct 24 19:34:41 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 24 19:34:41 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 24 19:34:41 vlad kernel: [] ? warn_slowpath_common+0x78/0x8d Oct 24 19:34:41 vlad kernel: [] ? dev_watchdog+0x0/0x188 Oct 24 19:34:41 vlad kernel: [] ? warn_slowpath_fmt+0x56/0x5e Oct 24 19:34:41 vlad kernel: [] ? hid_input_report+0x1bc/0x1d9 Oct 24 19:34:41 vlad kernel: [] ? netdev_drivername+0x3b/0x40 Oct 24 19:34:41 vlad kernel: [] ? dev_watchdog+0x104/0x188 Oct 24 19:34:41 vlad kernel: [] ? wq_worker_waking_up+0x8/0x27 Oct 24 19:34:41 vlad kernel: [] ? run_timer_softirq+0x14e/0x1c8 Oct 24 19:34:41 vlad kernel: [] ? __do_softirq+0x6b/0xe4 Oct 24 19:34:41 vlad kernel: [] ? tick_dev_program_event+0x2d/0xe9 Oct 24 19:34:41 vlad kernel: [] ? call_softirq+0x1a/0x24 Oct 24 19:34:41 vlad kernel: [] ? do_softirq+0x2f/0x61 Oct 24 19:34:41 vlad kernel: [] ? irq_exit+0x34/0x71 Oct 24 19:34:41 vlad kernel: [] ? smp_apic_timer_interrupt+0x71/0x7b Oct 24 19:34:41 vlad kernel: [] ? apic_timer_interrupt+0x13/0x20 Oct 24 19:34:41 vlad kernel: [] ? default_idle+0x1c/0x2f Oct 24 19:34:41 vlad kernel: [] ? cpu_idle+0x32/0x67 Oct 24 19:34:41 vlad kernel: [] ? start_kernel+0x301/0x30d Oct 24 19:34:41 vlad kernel: [] ? __reserve_early+0xa4/0xba Oct 24 19:34:41 vlad kernel: [] ? x86_64_start_kernel+0xe8/0xee Oct 24 19:34:41 vlad kernel: ---[ end trace 97448884c23863b0 ]--- Oct 22 21:10:50 vlad kernel: ------------[ cut here ]------------ Oct 22 21:10:50 vlad kernel: WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0xfa/0x18a() Oct 22 21:10:50 vlad kernel: Hardware name: System Product Name Oct 22 21:10:50 vlad kernel: NETDEV WATCHDOG: eth2 (via-rhine): transmit queue 0 timed out Oct 22 21:10:50 vlad kernel: Modules linked in: tcp_diag inet_diag loop cpufreq_userspace nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 xfs exportfs btrfs zlib_deflate hwmon_vid powernow_k8 sbp2 ieee1394 ide_generic ide_gd_mod ide_cd_mod evdev pcspkr i2c_viapro k8temp i2c_core hwmon pl2303 usbhid usbserial 8250_pnp 8250 serial_core button processor dm_mirror dm_region_hash dm_log dm_snapshot dm_mod usb_storage usb_libusual raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 md_mod sg sr_mod cdrom ehci_hcd via82cxxx via_rhine uhci_hcd mii sd_mod usbcore pata_via ide_pci_generic ide_core floppy thermal fan unix [last unloaded: scsi_wait_scan] Oct 22 21:10:50 vlad kernel: Pid: 0, comm: swapper Not tainted 2.6.34-81023-g2ebc346 #1 Oct 22 21:10:50 vlad kernel: Call Trace: Oct 22 21:10:50 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 22 21:10:50 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 22 21:10:50 vlad kernel: [] ? warn_slowpath_common+0x77/0xa3 Oct 22 21:10:50 vlad kernel: [] ? dev_watchdog+0x0/0x18a Oct 22 21:10:50 vlad kernel: [] ? warn_slowpath_fmt+0x51/0x59 Oct 22 21:10:50 vlad kernel: [] ? activate_task+0x41/0x50 Oct 22 21:10:50 vlad kernel: [] ? netdev_drivername+0x3b/0x40 Oct 22 21:10:50 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 22 21:10:50 vlad kernel: [] ? delayed_work_timer_fn+0x0/0x13 Oct 22 21:10:50 vlad kernel: [] ? run_timer_softirq+0x146/0x19c Oct 22 21:10:50 vlad kernel: [] ? __do_softirq+0x6b/0xe4 Oct 22 21:10:50 vlad kernel: [] ? tick_dev_program_event+0x2d/0xe9 Oct 22 21:10:50 vlad kernel: [] ? call_softirq+0x1a/0x24 Oct 22 21:10:50 vlad kernel: [] ? do_softirq+0x2f/0x61 Oct 22 21:10:50 vlad kernel: [] ? irq_exit+0x34/0x6f Oct 22 21:10:50 vlad kernel: [] ? smp_apic_timer_interrupt+0x71/0x7b Oct 22 21:10:50 vlad kernel: [] ? apic_timer_interrupt+0x13/0x20 Oct 22 21:10:50 vlad kernel: [] ? default_idle+0x1a/0x2d Oct 22 21:10:50 vlad kernel: [] ? cpu_idle+0x2e/0x65 Oct 22 21:10:50 vlad kernel: [] ? start_kernel+0x30b/0x317 Oct 22 21:10:50 vlad kernel: [] ? __reserve_early+0xa4/0xba Oct 22 21:10:50 vlad kernel: [] ? x86_64_start_kernel+0xe5/0xeb Oct 22 21:10:50 vlad kernel: ---[ end trace 0dea117d671a2125 ]--- Oct 22 21:10:50 vlad kernel: eth2: Transmit timed out, status 0003, PHY status 786d, resetting... Oct 22 21:10:50 vlad kernel: eth2: link up, 100Mbps, full-duplex, lpa 0xCDE1 Oct 22 21:10:54 vlad kernel: eth2: Transmit timed out, status 0003, PHY status 786d, resetting... Oct 22 21:10:54 vlad kernel: eth2: link up, 100Mbps, full-duplex, lpa 0xCDE1 Oct 21 21:35:40 vlad kernel: ------------[ cut here ]------------ Oct 21 21:35:40 vlad kernel: WARNING: at net/sched/sch_generic.c:256 dev_watchdog+0xfa/0x18a() Oct 21 21:35:40 vlad kernel: Hardware name: System Product Name Oct 21 21:35:40 vlad kernel: NETDEV WATCHDOG: eth2 (via-rhine): transmit queue 0 timed out Oct 21 21:35:40 vlad kernel: Modules linked in: tcp_diag inet_diag loop cpufreq_userspace nfsd nfs lockd nfs_acl auth_rpcgss sunrpc bridge stp llc ipv6 xfs exportfs btrfs zlib_deflate hwmon_vid powernow_k8 sbp2 ieee1394 ide_generic ide_gd_mod ide_cd_mod evdev pcspkr k8temp hwmon i2c_viapro pl2303 i2c_core usbserial usbhid 8250_pnp 8250 serial_core processor button dm_mirror dm_region_hash dm_log dm_snapshot dm_mod usb_storage usb_libusual raid456 async_pq async_xor xor async_memcpy async_raid6_recov raid6_pq async_tx raid1 md_mod sg sr_mod cdrom ehci_hcd via82cxxx uhci_hcd via_rhine ide_pci_generic mii usbcore pata_via sd_mod ide_core floppy thermal fan unix [last unloaded: scsi_wait_scan] Oct 21 21:35:40 vlad kernel: Pid: 0, comm: swapper Not tainted 2.6.34-81023-g2ebc346 #1 Oct 21 21:35:40 vlad kernel: Call Trace: Oct 21 21:35:40 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 21 21:35:40 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 21 21:35:40 vlad kernel: [] ? warn_slowpath_common+0x77/0xa3 Oct 21 21:35:40 vlad kernel: [] ? dev_watchdog+0x0/0x18a Oct 21 21:35:40 vlad kernel: [] ? warn_slowpath_fmt+0x51/0x59 Oct 21 21:35:40 vlad kernel: [] ? activate_task+0x41/0x50 Oct 21 21:35:40 vlad kernel: [] ? br_transmit_config+0x129/0x130 [bridge] Oct 21 21:35:40 vlad kernel: [] ? netdev_drivername+0x3b/0x40 Oct 21 21:35:40 vlad kernel: [] ? dev_watchdog+0xfa/0x18a Oct 21 21:35:40 vlad kernel: [] ? br_config_bpdu_generation+0x41/0x5e [bridge] Oct 21 21:35:40 vlad kernel: [] ? run_timer_softirq+0x146/0x19c Oct 21 21:35:40 vlad kernel: [] ? __do_softirq+0x6b/0xe4 Oct 21 21:35:40 vlad kernel: [] ? tick_dev_program_event+0x2d/0xe9 Oct 21 21:35:40 vlad kernel: [] ? call_softirq+0x1a/0x24 Oct 21 21:35:40 vlad kernel: [] ? do_softirq+0x2f/0x61 Oct 21 21:35:40 vlad kernel: [] ? irq_exit+0x34/0x6f Oct 21 21:35:40 vlad kernel: [] ? smp_apic_timer_interrupt+0x71/0x7b Oct 21 21:35:40 vlad kernel: [] ? apic_timer_interrupt+0x13/0x20 Oct 21 21:35:40 vlad kernel: [] ? default_idle+0x1a/0x2d Oct 21 21:35:40 vlad kernel: [] ? cpu_idle+0x2e/0x65 Oct 21 21:35:40 vlad kernel: [] ? start_kernel+0x30b/0x317 Oct 21 21:35:40 vlad kernel: [] ? __reserve_early+0xa4/0xba Oct 21 21:35:40 vlad kernel: [] ? x86_64_start_kernel+0xe5/0xeb Oct 21 21:35:40 vlad kernel: ---[ end trace cd249e97d2f719b2 ]--- Oct 21 21:35:40 vlad kernel: eth2: Transmit timed out, status 0003, PHY status 786d, resetting... Oct 21 21:35:40 vlad kernel: eth2: link up, 100Mbps, full-duplex, lpa 0xCDE1 hrm@vlad:~ $ cat /proc/interrupts CPU0 0: 26 IO-APIC-edge timer 1: 2 IO-APIC-edge i8042 6: 291 IO-APIC-edge floppy 7: 1 IO-APIC-edge 9: 0 IO-APIC-fasteoi acpi 12: 3 IO-APIC-edge i8042 14: 0 IO-APIC-edge ide0 15: 0 IO-APIC-edge ide1 20: 40 IO-APIC-fasteoi uhci_hcd:usb1 21: 46347 IO-APIC-fasteoi ahci, uhci_hcd:usb3 22: 3245 IO-APIC-fasteoi uhci_hcd:usb2, ehci_hcd:usb5 23: 19459767 IO-APIC-fasteoi uhci_hcd:usb4, eth2 28: 1838365 IO-APIC-fasteoi ahci NMI: 0 Non-maskable interrupts LOC: 426837 Local timer interrupts SPU: 0 Spurious interrupts PMI: 0 Performance monitoring interrupts PND: 0 Performance pending work THR: 0 Threshold APIC interrupts MCE: 0 Machine check exceptions MCP: 13 Machine check polls ERR: 1 MIS: 0 -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Python is executable pseudocode; perl --- is executable line-noise.