From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751303AbcGWP3n (ORCPT ); Sat, 23 Jul 2016 11:29:43 -0400 Received: from forward7p.cmail.yandex.net ([87.250.241.192]:58255 "EHLO forward7p.cmail.yandex.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751032AbcGWP3j (ORCPT ); Sat, 23 Jul 2016 11:29:39 -0400 X-Greylist: delayed 438 seconds by postgrey-1.27 at vger.kernel.org; Sat, 23 Jul 2016 11:29:39 EDT Authentication-Results: smtp14.mail.yandex.net; dkim=pass header.i=@forstwoof.ru X-Yandex-Suid-Status: 1 0,1 0,1 0 To: Jiang Liu , Thomas Gleixner Cc: linux-kernel@vger.kernel.org From: Foster Snowhill Subject: PROBLEM: Intel I210AT NIC resets while using PCI passthrough on ESXi (regression) Organization: ForstWoof Message-ID: Date: Sat, 23 Jul 2016 18:22:21 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Thunderbird/49.0a2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org [1.] One line summary of the problem: Intel I210AT NIC resets while using PCI passthrough on ESXi (regression) [2.] Full description of the problem/report: Intel I210AT NIC running on the latest (4.7-rc7) kernel revision suffers a "transmit queue 0 timed out" issue, which makes the NIC unusable. Each time this happens, the adapter is reset. To trigger the bug, bring up the adapter and attempt to transmit something. In my case it was enough to configure IPv4 on it and ping something. The more data is attempted to be transmitted, the more often the abovementioned error occurs. The system runs on a VMware ESXi 6.0 hypervisor and the NIC is attached to the Linux VM via PCI passthrough. BIOS and ESXi are all latest stable versions as of June 23rd, 2016. A bisect was performed, which pointed to commit 52f518a3a7c2f80551a38d38be28bc9f335e713c (x86/MSI: Use hierarchical irqdomains to manage MSI interrupts) being the first bad one. Full bisect log is below in part [9.]. It was noted that sometimes the bug does not reproduce the first time any given kernel version is booted, however subsequent boots reproduce the issue. The following appears in the kernel log when the bug occurs (also appended relevant information, MAC addresses edited out). [ 2.099565] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k [ 2.100488] igb: Copyright (c) 2007-2014 Intel Corporation. [ 2.106832] igb 0000:1b:00.0: enabling device (0000 -> 0002) [ 2.146246] igb 0000:1b:00.0: added PHC on eth1 [ 2.147211] igb 0000:1b:00.0: Intel(R) Gigabit Ethernet Network Connection [ 2.148468] igb 0000:1b:00.0: eth1: (PCIe:5.0Gb/s:unknown) xx:xx:xx:xx:xx:xx [ 2.149846] igb 0000:1b:00.0: eth1: PBA No: 001300-000 [ 2.150900] igb 0000:1b:00.0: Using MSI-X interrupts. 1 rx queue(s), 1 tx queue(s) [ 2.502960] igb 0000:1b:00.0 eth-wan2: renamed from eth1 [ 15.836869] igb 0000:1b:00.0 eth-wan2: igb: eth-wan2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [ 20.848027] ------------[ cut here ]------------ [ 20.848033] WARNING: CPU: 0 PID: 0 at /home/kernel/COD/linux/net/sched/sch_generic.c:272 dev_watchdog+0x22d/0x230 [ 20.848034] NETDEV WATCHDOG: eth-wan2 (igb): transmit queue 0 timed out [ 20.848035] Modules linked in: vmw_vsock_vmci_transport vsock fuse intel_powerclamp coretemp intel_rapl_perf ppdev vmw_balloon joydev nls_utf8 nls_cp437 vfat fat efi_pstore efivars serio_raw sg battery parport_pc parport shpchp i2c_piix4 vmw_vmci acpi_cpufreq ac tpm_tis tpm evdev sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi efivarfs autofs4 ext4 crc16 jbd2 mbcache btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod hid_generic usbhid hid sr_mod cdrom crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel jitterentropy_rng sd_mod hmac drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd psmouse [ 20.848061] igb dca ptp pps_core i2c_algo_bit ata_generic vmxnet3 uhci_hcd ehci_pci ehci_hcd vmwgfx vmw_pvscsi ata_piix ahci libahci usbcore usb_common ttm libata drm_kms_helper scsi_mod drm fjes button [ 20.848069] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.7.0-040700rc7-generic #201607110032 [ 20.848070] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1506250318 06/25/2015 [ 20.848071] 0000000000000286 223eaef37d48856a ffffffff81321265 ffff88003c603e10 [ 20.848072] 0000000000000000 ffffffff81078a3e 0000000000000000 ffff88003c603e68 [ 20.848073] ffff88003434c000 0000000000000000 ffff880034cf1940 0000000000000008 [ 20.848075] Call Trace: [ 20.848076] [] ? dump_stack+0x5c/0x77 [ 20.848082] [] ? __warn+0xbe/0xe0 [ 20.848084] [] ? warn_slowpath_fmt+0x5f/0x80 [ 20.848086] [] ? dev_watchdog+0x22d/0x230 [ 20.848087] [] ? qdisc_rcu_free+0x40/0x40 [ 20.848089] [] ? call_timer_fn+0x30/0x120 [ 20.848090] [] ? qdisc_rcu_free+0x40/0x40 [ 20.848091] [] ? run_timer_softirq+0x23c/0x2f0 [ 20.848094] [] ? __do_softirq+0xf8/0x28e [ 20.848095] [] ? irq_exit+0x9b/0xa0 [ 20.848096] [] ? smp_apic_timer_interrupt+0x3e/0x50 [ 20.848097] [] ? apic_timer_interrupt+0x82/0x90 [ 20.848098] [] ? native_safe_halt+0x2/0x10 [ 20.848102] [] ? default_idle+0x18/0xc0 [ 20.848103] [] ? cpu_startup_entry+0x2d9/0x330 [ 20.848106] [] ? start_kernel+0x44f/0x46f [ 20.848108] [] ? early_idt_handler_array+0x120/0x120 [ 20.848109] [] ? x86_64_start_kernel+0x14f/0x173 [ 20.848110] ---[ end trace 41fb9507befdccac ]--- [ 20.848125] igb 0000:1b:00.0 eth-wan2: Reset adapter [ 24.836669] igb 0000:1b:00.0 eth-wan2: igb: eth-wan2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX [4.] Kernel information [4.1.] Kernel version (from /proc/version): Linux version 4.7.0-040700rc7-generic (kernel@gloin) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1) ) #201607110032 SMP Mon Jul 11 04:34:25 UTC 2016 [5.] Most recent kernel that did not have the bug: Version 4.1 in mainline was the most recent kernel version without the bug. [8.] Environment [8.5.] PCI information ('lspci -vvv' as root) 1b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03) DeviceName: pciPassthru1 Subsystem: ASUSTeK Computer Inc. I210 Gigabit Network Connection Physical Slot: 256 Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR-