* 2.6.11-rc5 and 2.6.12: cannot transmit anything
@ 2005-07-25 5:17 Denis Vlasenko
2005-07-25 5:28 ` David S. Miller
0 siblings, 1 reply; 4+ messages in thread
From: Denis Vlasenko @ 2005-07-25 5:17 UTC (permalink / raw)
To: linux-kernel, linux-net; +Cc: David S. Miller, Jeff Garzik
[resend. Did not reach mailing lists, most probably due
to KMail's unstoppable desire to use base64 encoding :)]
Hi folks,
I reported earlied that around linux-2.6.11-rc5 my home box sometimes
does not want to send anything over ethetnet. That report is repeated below
sig.
I finally managed to nail down where this happens.
I instrumented sch_generic.c to trace what happens with packets
to be sent over interface named "if".
On 'good' boot, I see
2005-07-12_17:26:29.72158 kern.info: qdisc_restart: start
2005-07-12_17:26:29.72164 kern.info: qdisc_restart: skb!=NULL
2005-07-12_17:26:29.72166 kern.info: qdisc_restart: if !netif_queue_stopped...
2005-07-12_17:26:29.72167 kern.info: qdisc_restart: ...hard_start_xmit
in the log, on 'bad' one only "qdisc_restart: start".
Below is first report and instrumented part of sch_generic.c.
--
vda
Subject: linux-2.6.11-rc5: mysterious loss of tx
My home box has onboard via-rhine NIC.
Several days ago my father called me and said that
it does not send anything (tcpdump shows only rx'ed pkts
despite pings being attempted etc). I did not investigate
then.
Yesterday I've seen it myself. I bumped up ethtool msglvl.
Looks like via-rhine's hard_start_xmit was not called at all
from network core code! (I did not see debug printks from
rhine's hard_stat_xmit routine)
Whatever I tried (ifconfig down/up, reinit IP config from scratch),
nothing helped. No tx whatsoever was attempted by kernel, it seems.
Reboot 'fixed' things.
It hever happened on the same hardware before I switched to rc5.
int qdisc_restart(struct net_device *dev)
{
struct Qdisc *q = dev->qdisc;
struct sk_buff *skb;
int track = (dev->name[0]=='i' && dev->name[1]=='f' && dev->name[2]=='\0');
//'via rhine bug':
//I see ONLY "qdisc_restart: start",
//but not any of below msgs.
//On 'good' boots, it looks like this:
//...
//2005-07-12_17:26:29.72158 kern.info: qdisc_restart: start
//2005-07-12_17:26:29.72164 kern.info: qdisc_restart: skb!=NULL
//2005-07-12_17:26:29.72166 kern.info: qdisc_restart: if !netif_queue_stopped...
//2005-07-12_17:26:29.72167 kern.info: qdisc_restart: ...hard_start_xmit
//...
if(track) { printk("qdisc_restart: start\n"); }
/* Dequeue packet */
if ((skb = q->dequeue(q)) != NULL) {
if(track) { printk("qdisc_restart: skb!=NULL\n"); }
unsigned nolock = (dev->features & NETIF_F_LLTX);
/*
* When the driver has LLTX set it does its own locking
* in start_xmit. No need to add additional overhead by
* locking again. These checks are worth it because
* even uncongested locks can be quite expensive.
* The driver can do trylock like here too, in case
* of lock congestion it should return -1 and the packet
* will be requeued.
*/
if (!nolock) {
if (!spin_trylock(&dev->xmit_lock)) {
collision:
if(track) { printk("qdisc_restart: collision\n"); }
/* So, someone grabbed the driver. */
/* It may be transient configuration error,
when hard_start_xmit() recurses. We detect
it by checking xmit owner and drop the
packet when deadloop is detected.
*/
if (dev->xmit_lock_owner == smp_processor_id()) {
kfree_skb(skb);
if (net_ratelimit())
printk(KERN_DEBUG "Dead loop on netdevice %s, fix it urgently!\n", dev->name);
return -1;
}
__get_cpu_var(netdev_rx_stat).cpu_collision++;
goto requeue;
}
/* Remember that the driver is grabbed by us. */
dev->xmit_lock_owner = smp_processor_id();
}
{
/* And release queue */
spin_unlock(&dev->queue_lock);
//vda
if(track) { printk("qdisc_restart: if !netif_queue_stopped...\n"); }
if (!netif_queue_stopped(dev)) {
int ret;
if (netdev_nit)
dev_queue_xmit_nit(skb, dev);
if(track) { printk("qdisc_restart: ...hard_start_xmit\n"); }
ret = dev->hard_start_xmit(skb, dev);
if (ret == NETDEV_TX_OK) {
if (!nolock) {
dev->xmit_lock_owner = -1;
spin_unlock(&dev->xmit_lock);
}
spin_lock(&dev->queue_lock);
return -1;
}
if (ret == NETDEV_TX_LOCKED && nolock) {
spin_lock(&dev->queue_lock);
goto collision;
}
}
/* NETDEV_TX_BUSY - we need to requeue */
/* Release the driver */
if (!nolock) {
dev->xmit_lock_owner = -1;
spin_unlock(&dev->xmit_lock);
}
spin_lock(&dev->queue_lock);
q = dev->qdisc;
}
/* Device kicked us out :(
This is possible in three cases:
0. driver is locked
1. fastroute is enabled
2. device cannot determine busy state
before start of transmission (f.e. dialout)
3. device is buggy (ppp)
*/
requeue:
q->ops->requeue(skb, q);
netif_schedule(dev);
return 1;
}
BUG_ON((int) q->q.qlen < 0);
return q->q.qlen;
}
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.11-rc5 and 2.6.12: cannot transmit anything
2005-07-25 5:17 2.6.11-rc5 and 2.6.12: cannot transmit anything Denis Vlasenko
@ 2005-07-25 5:28 ` David S. Miller
2005-07-25 5:53 ` Denis Vlasenko
0 siblings, 1 reply; 4+ messages in thread
From: David S. Miller @ 2005-07-25 5:28 UTC (permalink / raw)
To: vda; +Cc: linux-kernel, linux-net, jgarzik
Probably your link is never coming up. We won't send packets
over the wire unless the device is in the link-up state.
However, if ->dequeue() is returning NULL, there really aren't
any packets in the device queue to be sent.
If you want, add more tracing to pfifo_fast_dequeue() since
that's almost certainly which queueing discipline is hooked
up to your VIA Rhine device as it's the default.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.11-rc5 and 2.6.12: cannot transmit anything
2005-07-25 5:28 ` David S. Miller
@ 2005-07-25 5:53 ` Denis Vlasenko
0 siblings, 0 replies; 4+ messages in thread
From: Denis Vlasenko @ 2005-07-25 5:53 UTC (permalink / raw)
To: David S. Miller; +Cc: linux-kernel, linux-net, jgarzik
On Monday 25 July 2005 08:28, David S. Miller wrote:
>
> Probably your link is never coming up. We won't send packets
> over the wire unless the device is in the link-up state.
>
> However, if ->dequeue() is returning NULL, there really aren't
> any packets in the device queue to be sent.
>
> If you want, add more tracing to pfifo_fast_dequeue() since
> that's almost certainly which queueing discipline is hooked
> up to your VIA Rhine device as it's the default.
Will do.
--
vda
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: 2.6.11-rc5 and 2.6.12: cannot transmit anything
@ 2005-07-31 10:59 Denis Vlasenko
0 siblings, 0 replies; 4+ messages in thread
From: Denis Vlasenko @ 2005-07-31 10:59 UTC (permalink / raw)
To: David S. Miller; +Cc: linux-kernel, linux-net, jgarzik
[-- Attachment #1: Type: text/plain, Size: 4255 bytes --]
On Monday 25 July 2005 08:17, Denis Vlasenko wrote:
> I reported earlied that around linux-2.6.11-rc5 my home box sometimes
> does not want to send anything over ethernet. That report is repeated below
> sig.
>
> I finally managed to nail down where this happens.
> I instrumented sch_generic.c to trace what happens with packets
> to be sent over interface named "if".
>
> On 'good' boot, I see
>
> 2005-07-12_17:26:29.72158 kern.info: qdisc_restart: start
> 2005-07-12_17:26:29.72164 kern.info: qdisc_restart: skb!=NULL
> 2005-07-12_17:26:29.72166 kern.info: qdisc_restart: if !netif_queue_stopped...
> 2005-07-12_17:26:29.72167 kern.info: qdisc_restart: ...hard_start_xmit
>
> in the log, on 'bad' one only "qdisc_restart: start".
On Monday 25 July 2005 08:28, David S. Miller wrote:
> Probably your link is never coming up. We won't send packets
> over the wire unless the device is in the link-up state.
>
> However, if ->dequeue() is returning NULL, there really aren't
> any packets in the device queue to be sent.
>
> If you want, add more tracing to pfifo_fast_dequeue() since
> that's almost certainly which queueing discipline is hooked
> up to your VIA Rhine device as it's the default.
I have move info.
kernel log of "ping 1.1.4.1 -i 0.01" when everything is good:
...
2005-07-30_22:30:35.00559 kern.info: pfifo_fast_enqueue returns 0
2005-07-30_22:30:35.00563 kern.info: qdisc_restart: start
2005-07-30_22:30:35.00568 kern.info: pfifo_fast_dequeue returns a skb
2005-07-30_22:30:35.00570 kern.info: qdisc_restart: skb!=NULL
2005-07-30_22:30:35.00571 kern.info: qdisc_restart: start
2005-07-30_22:30:35.00573 kern.info: pfifo_fast_dequeue returns NULL
2005-07-30_22:30:35.01458 kern.info: pfifo_fast_enqueue returns 0
2005-07-30_22:30:35.01460 kern.info: qdisc_restart: start
2005-07-30_22:30:35.01462 kern.info: pfifo_fast_dequeue returns a skb
2005-07-30_22:30:35.01463 kern.info: qdisc_restart: skb!=NULL
2005-07-30_22:30:35.01468 kern.info: qdisc_restart: start
2005-07-30_22:30:35.01470 kern.info: pfifo_fast_dequeue returns NULL
2005-07-30_22:30:35.02358 kern.info: pfifo_fast_enqueue returns 0
2005-07-30_22:30:35.02360 kern.info: qdisc_restart: start
...
Kernel log when 'no tx syndrome' is in effect:
2005-07-30_21:28:25.15338 kern.info: qdisc_restart: start
2005-07-30_21:28:25.16438 kern.info: qdisc_restart: start
2005-07-30_21:28:25.17538 kern.info: qdisc_restart: start
2005-07-30_21:28:25.18638 kern.info: qdisc_restart: start
2005-07-30_21:28:25.19738 kern.info: qdisc_restart: start
2005-07-30_21:28:25.20837 kern.info: qdisc_restart: start
2005-07-30_21:28:25.21937 kern.info: qdisc_restart: start
2005-07-30_21:28:25.23037 kern.info: qdisc_restart: start
2005-07-30_21:28:25.24137 kern.info: qdisc_restart: start
2005-07-30_21:28:25.25237 kern.info: qdisc_restart: start
2005-07-30_21:28:25.26337 kern.info: qdisc_restart: start
2005-07-30_21:28:25.27436 kern.info: qdisc_restart: start
2005-07-30_21:28:25.28536 kern.info: qdisc_restart: start
I instrumented sch_generic.c as follows:
int qdisc_restart(struct net_device *dev)
{
struct Qdisc *q = dev->qdisc;
struct sk_buff *skb;
int track = (dev->name[0]=='i' && dev->name[1]=='f' && dev->name[2]=='\0');
if(track) { printk("qdisc_restart: start\n"); }
/* Dequeue packet */
if ((skb = q->dequeue(q)) != NULL) {
if(track) { printk("qdisc_restart: skb!=NULL\n"); }
...
static struct sk_buff *
pfifo_fast_dequeue(struct Qdisc* qdisc)
{
int prio;
struct sk_buff_head *list = qdisc_priv(qdisc);
struct sk_buff *skb;
for (prio = 0; prio < 3; prio++, list++) {
skb = __skb_dequeue(list);
if (skb) {
qdisc->q.qlen--;
printk("pfifo_fast_dequeue returns a skb\n");
return skb;
}
}
printk("pfifo_fast_dequeue returns NULL\n");
return NULL;
}
Since I don't see "pfifo_fast_dequeue returns..." messages,
it looks like interface "if" doesn't use pfifo_fast qdisc... ?!
I positively sure I do not use any shaping on that box,
and all interfaces ought to use default qdisc. Strange.
Will collect "ip l" output when it will happen next time.
Instrumented source file is attached.
--
vda
[-- Attachment #2: sch_generic.c.bz2 --]
[-- Type: application/x-bzip2, Size: 5054 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2005-07-31 10:59 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2005-07-25 5:17 2.6.11-rc5 and 2.6.12: cannot transmit anything Denis Vlasenko
2005-07-25 5:28 ` David S. Miller
2005-07-25 5:53 ` Denis Vlasenko
2005-07-31 10:59 Denis Vlasenko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®