4.9.28: VIA velocity hang: how to find cause?

4.9.28: VIA velocity hang: how to find cause?

[Udo van den Heuvel]

Hello,

I noticed multiple times that VIA velocity interface stop functioning
without reason but is revived after and `ifdown ethX; ifup ethX`.
No data is transferred until the `fix` is applied.
No errors are shown in dmesg or /var/log/messages w.r.t. the interface.
How to find the root cause?

This is with kernel 4.9.28.

Kind regards,
Udo

Re: 4.9.28: VIA velocity hang: how to find cause?

[Udo van den Heuvel]

Hello,

This issue occurs on VIA Epia M910E.
Both interfaces suffer from this.
Setting msglvl with ethtool has no effect on the info the driver
provides on any issues.
So how to proceed from here?
If this is a software issue, which I cannot determine, the driver is
useless in its present state.

Kind regards,
Udo

On 23-05-17 18:07, Udo van den Heuvel wrote:
> Hello,
> 
> I noticed multiple times that VIA velocity interface stop functioning
> without reason but is revived after and `ifdown ethX; ifup ethX`.
> No data is transferred until the `fix` is applied.
> No errors are shown in dmesg or /var/log/messages w.r.t. the interface.
> How to find the root cause?
> 
> This is with kernel 4.9.28.
> 
> Kind regards,
> Udo
> 

Re: 4.9.28: VIA velocity hang: how to find cause?

[Udo van den Heuvel]

Hello,

Forcing the VIA Velocity interface to 1000 Mbps FDX appears to help a
bit but still the connection stops transferring data after a while.
I added some printk's and in or after velocity_tx_service it ends.

Where should I focus with my debug printk's?

It appears I can reproduce the issue by running a `find /` over ssh to
the box. So perhaps it is an interrupt related issue?

Please help find the root cause.

Kind regards,
Udo

On 24-05-17 15:04, Udo van den Heuvel wrote:
> Hello,
> 
> This issue occurs on VIA Epia M910E.
> Both interfaces suffer from this.
> Setting msglvl with ethtool has no effect on the info the driver
> provides on any issues.
> So how to proceed from here?
> If this is a software issue, which I cannot determine, the driver is
> useless in its present state.
> 
> Kind regards,
> Udo
> 
> On 23-05-17 18:07, Udo van den Heuvel wrote:
>> Hello,
>>
>> I noticed multiple times that VIA velocity interface stop functioning
>> without reason but is revived after and `ifdown ethX; ifup ethX`.
>> No data is transferred until the `fix` is applied.
>> No errors are shown in dmesg or /var/log/messages w.r.t. the interface.
>> How to find the root cause?
>>
>> This is with kernel 4.9.28.
>>
>> Kind regards,
>> Udo
>>
> 

Re: 4.9.28: VIA velocity hang: how to find cause?

[Alan Cox]

On Thu, 25 May 2017 13:52:08 +0200
Udo van den Heuvel <udovdh@xs4all.nl> wrote:

> Hello,
> 
> Forcing the VIA Velocity interface to 1000 Mbps FDX appears to help a
> bit but still the connection stops transferring data after a while.
> I added some printk's and in or after velocity_tx_service it ends.
> 
> Where should I focus with my debug printk's?
> 
> It appears I can reproduce the issue by running a `find /` over ssh to
> the box. So perhaps it is an interrupt related issue?
> 
> Please help find the root cause.
> 

netdev@vger.kernel.org would be a better place to ask

But some things to play with

1. See if an scp of a very large file from a local machine can trigger it

2. See if you can trigger it with a small number of ping -f commands

3. Try both of them the other way around to see if its rx or tx related

Also see if you can duplicate the problem on another machine - just in
case. Saves spending days chasing some kind of insane hardware
compatibility problem 8)

My first guess given that more speed helps would be that something is
mishandling the ring buffer filling on receive (and that in turn causes
tcp to back off) so I would look at logging ring full events. Also dump
the device state after it hangs and see what the hardware thinks is going
on (eg IRQ's pending, rings full, errors logged, link state ....)

tcpdump might also give you some clues providing it doesn't change the
performance too much and hide the problem. You'll get to see if there are
any dropped frames/back offs/where it gets stuck.

Alan