Re: [Stable-review] [22/68] ixgbe: prevent speculative processing of descriptors before ready

Re: [Stable-review] [22/68] ixgbe: prevent speculative processing of descriptors before ready

[miltonm]

----- Original Message Follows -----
From: Ben Hutchings <ben@decadent.org.uk>
To: Greg KH <gregkh@suse.de>
Cc: linux-kernel@vger.kernel.org, stable@kernel.org, Don
Skidmore <donald.c.skidmore@intel.com>, Milton Miller
<miltonm@bga.com>, "David S. Miller" <davem@davemloft.net>,
Anton Blanchard <anton@samba.org>,
akpm@linux-foundation.org,  torvalds@linux-foundation.org,
stable-review@kernel.org,  alan@lxorguk.ukuu.org.uk, Jeff
Kirsher <jeffrey.t.kirsher@intel.com>
Subject: Re: [Stable-review] [22/68] ixgbe: prevent
speculative processing of descriptors before ready
Date: Tue, 01 Mar 2011 02:14:33 +0000

> On Mon, 2011-02-28 at 08:22 -0800, Greg KH wrote:
> > 2.6.32-longterm review patch.  If anyone has any
> > objections, please let us know. 
> > ------------------
> > 
> > From: Milton Miller <miltonm@bga.com>
> > 
> > commit 3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a
> > upstream. 
> > The PowerPC architecture does not require loads to
> > independent bytes to be ordered without adding an
> > explicit barrier. 
> > In ixgbe_clean_rx_irq we load the status bit then load
> > the packet data. With packet split disabled if these
> > loads go out of order we get a stale packet, but we will
> > notice the bad sequence numbers and drop it. 
> > The problem occurs with packet split enabled where the
> > TCP/IP header and data are in different descriptors. If
> > the reads go out of order we may have data that doesn't
> match the TCP/IP header. Since we use hardware
> > checksumming this bad data is never verified and it
> makes it all the way to the application. [...]
> 
> Packet splitting should be completely disabled on the
> 82599, anyway. Greg, what happened to this patch:

This was initially observed on 82598 and is still needed
there.

milton

[22/68] ixgbe: prevent speculative processing of descriptors before ready

[Greg KH]

2.6.32-longterm review patch.  If anyone has any objections, please let us know.

------------------

From: Milton Miller <miltonm@bga.com>

commit 3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a upstream.

The PowerPC architecture does not require loads to independent bytes to be
ordered without adding an explicit barrier.

In ixgbe_clean_rx_irq we load the status bit then load the packet data.
With packet split disabled if these loads go out of order we get a
stale packet, but we will notice the bad sequence numbers and drop it.

The problem occurs with packet split enabled where the TCP/IP header and data
are in different descriptors. If the reads go out of order we may have data
that doesn't match the TCP/IP header. Since we use hardware checksumming this
bad data is never verified and it makes it all the way to the application.

This bug was found during stress testing and adding this barrier has been shown
to fix it.

Signed-off-by: Milton Miller <miltonm@bga.com>
Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: Don Skidmore <donald.c.skidmore@intel.com>
Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Cc: maximilian attems <max@stro.at>
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>

---
 drivers/net/ixgbe/ixgbe_main.c |    1 +
 1 file changed, 1 insertion(+)

--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -793,6 +793,7 @@ static bool ixgbe_clean_rx_irq(struct ix
 			break;
 		(*work_done)++;
 
+		rmb(); /* read descriptor and rx_buffer_info after status DD */
 		if (rx_ring->flags & IXGBE_RING_RX_PS_ENABLED) {
 			hdr_info = le16_to_cpu(ixgbe_get_hdr_info(rx_desc));
 			len = (hdr_info & IXGBE_RXDADV_HDRBUFLEN_MASK) >>

Re: [Stable-review] [22/68] ixgbe: prevent speculative processing of descriptors before ready

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 1862 bytes --]

On Mon, 2011-02-28 at 08:22 -0800, Greg KH wrote:
> 2.6.32-longterm review patch.  If anyone has any objections, please let us know.
> 
> ------------------
> 
> From: Milton Miller <miltonm@bga.com>
> 
> commit 3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a upstream.
> 
> The PowerPC architecture does not require loads to independent bytes to be
> ordered without adding an explicit barrier.
> 
> In ixgbe_clean_rx_irq we load the status bit then load the packet data.
> With packet split disabled if these loads go out of order we get a
> stale packet, but we will notice the bad sequence numbers and drop it.
> 
> The problem occurs with packet split enabled where the TCP/IP header and data
> are in different descriptors. If the reads go out of order we may have data
> that doesn't match the TCP/IP header. Since we use hardware checksumming this
> bad data is never verified and it makes it all the way to the application.
[...]

Packet splitting should be completely disabled on the 82599, anyway.
Greg, what happened to this patch:

commit a124339ad28389093ed15eca990d39c51c5736cc
Author: Don Skidmore <donald.c.skidmore@intel.com>
Date:   Tue Jan 18 22:53:47 2011 +0000

    ixgbe: fix for 82599 erratum on Header Splitting
    
    We have found a hardware erratum on 82599 hardware that can lead to
    unpredictable behavior when Header Splitting mode is enabled.  So
    we are no longer enabling this feature on affected hardware.
    
    Please see the 82599 Specification Update for more information.
    
    CC: stable@kernel.org
    Signed-off-by: Don Skidmore <donald.c.skidmore@intel.com>
    Tested-by: Stephen Ko <stephen.s.ko@intel.com>
    Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

Ben.

-- 
Ben Hutchings
Once a job is fouled up, anything done to improve it makes it worse.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Stable-review] [22/68] ixgbe: prevent speculative processing of descriptors before ready

[Greg KH]

On Tue, Mar 01, 2011 at 02:14:33AM +0000, Ben Hutchings wrote:
> On Mon, 2011-02-28 at 08:22 -0800, Greg KH wrote:
> > 2.6.32-longterm review patch.  If anyone has any objections, please let us know.
> > 
> > ------------------
> > 
> > From: Milton Miller <miltonm@bga.com>
> > 
> > commit 3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a upstream.
> > 
> > The PowerPC architecture does not require loads to independent bytes to be
> > ordered without adding an explicit barrier.
> > 
> > In ixgbe_clean_rx_irq we load the status bit then load the packet data.
> > With packet split disabled if these loads go out of order we get a
> > stale packet, but we will notice the bad sequence numbers and drop it.
> > 
> > The problem occurs with packet split enabled where the TCP/IP header and data
> > are in different descriptors. If the reads go out of order we may have data
> > that doesn't match the TCP/IP header. Since we use hardware checksumming this
> > bad data is never verified and it makes it all the way to the application.
> [...]
> 
> Packet splitting should be completely disabled on the 82599, anyway.
> Greg, what happened to this patch:
> 
> commit a124339ad28389093ed15eca990d39c51c5736cc
> Author: Don Skidmore <donald.c.skidmore@intel.com>
> Date:   Tue Jan 18 22:53:47 2011 +0000
> 
>     ixgbe: fix for 82599 erratum on Header Splitting
>     
>     We have found a hardware erratum on 82599 hardware that can lead to
>     unpredictable behavior when Header Splitting mode is enabled.  So
>     we are no longer enabling this feature on affected hardware.
>     
>     Please see the 82599 Specification Update for more information.
>     
>     CC: stable@kernel.org
>     Signed-off-by: Don Skidmore <donald.c.skidmore@intel.com>
>     Tested-by: Stephen Ko <stephen.s.ko@intel.com>
>     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>

It does not apply to the .32 kernel.

Care to provide a backport if you feel it should be there?

thanks,

greg k-h

Re: [Stable-review] [22/68] ixgbe: prevent speculative processing of descriptors before ready

[Jeff Kirsher]

[-- Attachment #1: Type: text/plain, Size: 2274 bytes --]

On Tue, 2011-03-01 at 12:46 -0800, Greg KH wrote:
> On Tue, Mar 01, 2011 at 02:14:33AM +0000, Ben Hutchings wrote:
> > On Mon, 2011-02-28 at 08:22 -0800, Greg KH wrote:
> > > 2.6.32-longterm review patch.  If anyone has any objections, please let us know.
> > > 
> > > ------------------
> > > 
> > > From: Milton Miller <miltonm@bga.com>
> > > 
> > > commit 3c945e5b3719bcc18c6ddd31bbcae8ef94f3d19a upstream.
> > > 
> > > The PowerPC architecture does not require loads to independent bytes to be
> > > ordered without adding an explicit barrier.
> > > 
> > > In ixgbe_clean_rx_irq we load the status bit then load the packet data.
> > > With packet split disabled if these loads go out of order we get a
> > > stale packet, but we will notice the bad sequence numbers and drop it.
> > > 
> > > The problem occurs with packet split enabled where the TCP/IP header and data
> > > are in different descriptors. If the reads go out of order we may have data
> > > that doesn't match the TCP/IP header. Since we use hardware checksumming this
> > > bad data is never verified and it makes it all the way to the application.
> > [...]
> > 
> > Packet splitting should be completely disabled on the 82599, anyway.
> > Greg, what happened to this patch:
> > 
> > commit a124339ad28389093ed15eca990d39c51c5736cc
> > Author: Don Skidmore <donald.c.skidmore@intel.com>
> > Date:   Tue Jan 18 22:53:47 2011 +0000
> > 
> >     ixgbe: fix for 82599 erratum on Header Splitting
> >     
> >     We have found a hardware erratum on 82599 hardware that can lead to
> >     unpredictable behavior when Header Splitting mode is enabled.  So
> >     we are no longer enabling this feature on affected hardware.
> >     
> >     Please see the 82599 Specification Update for more information.
> >     
> >     CC: stable@kernel.org
> >     Signed-off-by: Don Skidmore <donald.c.skidmore@intel.com>
> >     Tested-by: Stephen Ko <stephen.s.ko@intel.com>
> >     Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@intel.com>
> 
> It does not apply to the .32 kernel.
> 
> Care to provide a backport if you feel it should be there?
> 
> thanks,
> 
> greg k-h

Don is working on patch for .32 kernel, I hope to send it out no later
than tomorrow.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 490 bytes --]