Xen + linux 6.1.0-rc8, network to guest VM not working after commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423

Xen + linux 6.1.0-rc8, network to guest VM not working after commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423

[Sander Eikelenboom]

Hi Ross / Juergen,

I just updated my linux kernel to the latest of Linus his tree which included commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423.

Unfortunately when using this kernel I can't SSH anymore into the Xen guest I start, but I don't see any apparent failures either.
A straight revert of the commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a makes networking function normally again.

I have added some of the logging below, perhaps it at gives some idea off the state around the Xen network front and backend.

Any ideas or a test patch that I could run to shed some more light on what is going on ?

--
Sander




Some of the logging from dom0 dmesg:

[  149.520585] xen_bridge: port 1(vif1.0) entered blocking state
[  149.520594] xen_bridge: port 1(vif1.0) entered disabled state
[  149.520678] device vif1.0 entered promiscuous mode
[  151.221975] xen-blkback: backend/vbd/1/51712: using 1 queues, protocol 1 (x86_64-abi) persistent grants
[  151.601458] vif vif-1-0 vif1.0: Guest Rx ready
[  151.601476] xen_bridge: port 1(vif1.0) entered blocking state
[  151.601478] xen_bridge: port 1(vif1.0) entered forwarding state


output xenstore-ls regarding vif for the Guest:

     vif = ""
      1 = ""
       0 = ""
        bridge = "xen_bridge"
        feature-ctrl-ring = "1"
        feature-dynamic-multicast-control = "1"
        feature-gso-tcpv4 = "1"
        feature-gso-tcpv6 = "1"
        feature-ipv6-csum-offload = "1"
        feature-multicast-control = "1"
        feature-rx-copy = "1"
        feature-rx-flip = "0"
        feature-sg = "1"
        feature-split-event-channels = "1"
        feature-xdp-headroom = "1"
        frontend = "/local/domain/1/device/vif/0"
        frontend-id = "1"
        handle = "0"
        hotplug-status = "connected"
        ip = "192.168.1.6"
        mac = "00:16:3e:49:0e:fa"
        multi-queue-max-queues = "8"
        online = "1"
        script = "/etc/xen/scripts/vif-bridge"
        state = "4"
        type = "vif"

     vif = ""
      0 = ""
       backend = "/local/domain/0/backend/vif/1/0"
       backend-id = "0"
       event-channel-rx = "9"
       event-channel-tx = "8"
       feature-gso-tcpv4 = "1"
       feature-gso-tcpv6 = "1"
       feature-ipv6-csum-offload = "1"
       feature-rx-notify = "1"
       feature-sg = "1"
       handle = "0"
       mac = "00:16:3e:49:0e:fa"
       mtu = "1500"
       multi-queue-num-queues = "1"
       request-rx-copy = "1"
       rx-ring-ref = "524"
       state = "4"
       trusted = "1"
       tx-ring-ref = "523"
       xdp-headroom = "0"


ifconfig output for the guest interface on dom0 side:

vif1.0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
         ether fe:ff:ff:ff:ff:ff  txqueuelen 32  (Ethernet)
         RX packets 0  bytes 0 (0.0 B)
         RX errors 0  dropped 0  overruns 0  frame 0
         TX packets 49  bytes 2058 (2.0 KiB)
         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Re: Xen + linux 6.1.0-rc8, network to guest VM not working after commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423

[Andrew Cooper]

On 07/12/2022 21:42, Sander Eikelenboom wrote:
> Hi Ross / Juergen,
>
> I just updated my linux kernel to the latest of Linus his tree which
> included commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423.
>
> Unfortunately when using this kernel I can't SSH anymore into the Xen
> guest I start, but I don't see any apparent failures either.
> A straight revert of the commit
> ad7f402ae4f466647c3a669b8a6f3e5d4271c84a makes networking function
> normally again.
>
> I have added some of the logging below, perhaps it at gives some idea
> off the state around the Xen network front and backend.
>
> Any ideas or a test patch that I could run to shed some more light on
> what is going on ?

XSA-423 was buggy.  Fix and discussion at:

https://lore.kernel.org/xen-devel/681773dd-6264-63ac-a3b5-a9182b9e0cc1@suse.com/T/#t

~Andrew

Re: Xen + linux 6.1.0-rc8, network to guest VM not working after commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423

[Sander Eikelenboom]

On 08/12/2022 00:00, Andrew Cooper wrote:
> On 07/12/2022 21:42, Sander Eikelenboom wrote:
>> Hi Ross / Juergen,
>>
>> I just updated my linux kernel to the latest of Linus his tree which
>> included commit ad7f402ae4f466647c3a669b8a6f3e5d4271c84a fixing XSA-423.
>>
>> Unfortunately when using this kernel I can't SSH anymore into the Xen
>> guest I start, but I don't see any apparent failures either.
>> A straight revert of the commit
>> ad7f402ae4f466647c3a669b8a6f3e5d4271c84a makes networking function
>> normally again.
>>
>> I have added some of the logging below, perhaps it at gives some idea
>> off the state around the Xen network front and backend.
>>
>> Any ideas or a test patch that I could run to shed some more light on
>> what is going on ?
> 
> XSA-423 was buggy.  Fix and discussion at:
> 
> https://lore.kernel.org/xen-devel/681773dd-6264-63ac-a3b5-a9182b9e0cc1@suse.com/T/#t
> 
> ~Andrew

Thanks for the pointer Andrew, that fix works for me as well!
What a difference a few characters can make :)

--
Sander