Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Sat, 2 Dec 2006, Karsten Weiss wrote:

> On Sat, 2 Dec 2006, Christoph Anton Mitterer wrote:
> 
> > I found a severe bug mainly by fortune because it occurs very rarely.
> > My test looks like the following: I have about 30GB of testing data on
> 
> This sounds very familiar! One of the Linux compute clusters I
> administer at work is a 336 node system consisting of the
> following components:
> 
> * 2x Dual-Core AMD Opteron 275
> * Tyan S2891 mainboard
> * Hitachi HDS728080PLA380 harddisk
> * 4 GB RAM (some nodes have 8 GB) - intensively tested with
>   memtest86+
> * SUSE 9.3 x86_64 (kernel 2.6.11.4-21.14-smp) - But I've also
>   e.g. tried the latest openSUSE 10.2 RC1+ kernel 2.6.18.2-33 which
>   makes no difference.
> 
> We are running LS-Dyna on these machines and discovered a
> testcase which shows a similar data corruption. So I can
> confirm that the problem is for real an not a hardware defect
> of a single machine!

Last week we did some more testing with the following result:

We could not reproduce the data corruption anymore if we boot the machines 
with the kernel parameter "iommu=soft" i.e. if we use software bounce 
buffering instead of the hw-iommu. (As mentioned before, booting with 
mem=2g works fine, too, because this disables the iommu altogether.)

I.e. on these systems the data corruption only happens if the hw-iommu 
(PCI-GART) of the Opteron CPUs is in use.

Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
confirm this workaround, too.

Best regards,
Karsten

-- 
__________________________________________creating IT solutions
Dipl.-Inf. Karsten Weiss               science + computing ag
phone:    +49 7071 9457 452            Hagellocher Weg 73
teamline: +49 7071 9457 681            72070 Tuebingen, Germany
email:    knweiss@science-computing.de www.science-computing.de

amd64 iommu causing corruption? (was Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!)

[Chris Wedgwood]

On Mon, Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:

> We could not reproduce the data corruption anymore if we boot the
> machines with the kernel parameter "iommu=soft" i.e. if we use
> software bounce buffering instead of the hw-iommu. (As mentioned
> before, booting with mem=2g works fine, too, because this disables
> the iommu altogether.)

I can confirm this also seems to be the case for me, I'm still doing
more testing to confirm this.  But it would seem:

nforce4, transfer of a large mount of data with 4GB+ of RAM I get some
corruption.  This is present on both the nv SATA and also Sil 3112
connected drives.

Using iommu=soft so far seems to be working without any corruption.



I still need to do more testing on other machines which have less
memory (so the IOMMU won't be in use there either) and see if there
are problems there.

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1601 bytes --]

Karsten Weiss wrote:
> Last week we did some more testing with the following result:
>
> We could not reproduce the data corruption anymore if we boot the machines 
> with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> buffering instead of the hw-iommu. (As mentioned before, booting with 
> mem=2g works fine, too, because this disables the iommu altogether.)
>   
I can confirm this,...
booting with mem=2G => works fine,...

(all of the following tests were made with memory hole mapping=hardware
in the BIOS,.. so I could access my full ram):
booting with iommu=soft => works fine
booting with iommu=noagp => DOESN'T solve the error
booting with iommu=off => the system doesn't even boot and panics

When I set IOMMU to disabled in the BIOS the error is not solved-
I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
64MB),.. but it does not solve the problem.

Any ideas why iommu=disabled in the bios does not solve the issue?

> I.e. on these systems the data corruption only happens if the hw-iommu 
> (PCI-GART) of the Opteron CPUs is in use.
>   
1) And does this now mean that there's an error in the hardware (chipset
or CPU/memcontroller)?


> Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> confirm this workaround, too.
>   
Yes I can absolutely confirm this...
Do my additional tests help you?



Do you have any ideas why the issue doesn't occur (even with memhole
mapping=hardware in the bios and no iommu=soft at kernel command line)
when dma is disabled for the disks (or a slower dma mode is used)?


Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:18:21PM +0100, Christoph Anton Mitterer wrote:

> booting with iommu=soft => works fine
> booting with iommu=noagp => DOESN'T solve the error
> booting with iommu=off => the system doesn't even boot and panics

> When I set IOMMU to disabled in the BIOS the error is not solved-
> I tried to set bigger space for the IOMMU in the BIOS (256MB instead of
> 64MB),.. but it does not solve the problem.

> Any ideas why iommu=disabled in the bios does not solve the issue?

The kernel will still use the IOMMU if the BIOS doesn't set it up if
it can, check your dmesg for IOMMU strings, there might be something
printed to this effect.

> 1) And does this now mean that there's an error in the hardware
> (chipset or CPU/memcontroller)?

My guess is it's a kernel bug, I don't know for certain.  Perhaps we
shaould start making a more comprehensive list of affected kernels &
CPUs?

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Chris Wedgwood wrote:

> > Any ideas why iommu=disabled in the bios does not solve the issue?
> 
> The kernel will still use the IOMMU if the BIOS doesn't set it up if
> it can, check your dmesg for IOMMU strings, there might be something
> printed to this effect.

FWIW: As far as I understand the linux kernel code (I am no kernel 
developer so please correct me if I am wrong) the PCI dma mapping code is 
abstracted by struct dma_mapping_ops. I.e. there are currently four 
possible implementations for x86_64 (see linux-2.6/arch/x86_64/kernel/)

1. pci-nommu.c : no IOMMU at all (e.g. because you have < 4 GB memory)
   Kernel boot message: "PCI-DMA: Disabling IOMMU."

2. pci-gart.c : (AMD) Hardware-IOMMU.
   Kernel boot message: "PCI-DMA: using GART IOMMU" (this message
   first appeared in 2.6.16)

3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
   Kernel boot message: "PCI-DMA: Using software bounce buffering 
   for IO (SWIOTLB)"

4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
   This HW-IOMMU supports dma address mapping with memory proctection,
   etc.
   Kernel boot message: "PCI-DMA: Using Calgary IOMMU" (since 2.6.18!)

What all this means is that you can use "dmesg|grep ^PCI-DMA:" to see 
which implementation your kernel is currently using.

As far as our problem machines are concerned the "PCI-DMA: using GART 
IOMMU" case is broken (data corruption). But both "PCI-DMA: Disabling 
IOMMU" (trigged with mem=2g) and "PCI-DMA: Using software bounce buffering 
for IO (SWIOTLB)" (triggered with iommu=soft) are stable.

BTW: It would be really great if this area of the kernel would get some 
more and better documentation. The information at 
linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
read the code to get a *rough* idea what all the "iommu=" options 
actually do and how they interact.
 
> > 1) And does this now mean that there's an error in the hardware
> > (chipset or CPU/memcontroller)?
> 
> My guess is it's a kernel bug, I don't know for certain.  Perhaps we
> shaould start making a more comprehensive list of affected kernels &
> CPUs?

BTW: Did someone already open an official bug at 
http://bugzilla.kernel.org ?

Best regards,
Karsten

-- 
__________________________________________creating IT solutions
Dipl.-Inf. Karsten Weiss               science + computing ag
phone:    +49 7071 9457 452            Hagellocher Weg 73
teamline: +49 7071 9457 681            72070 Tuebingen, Germany
email:    knweiss@science-computing.de www.science-computing.de

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:

> FWIW: As far as I understand the linux kernel code (I am no kernel 
> developer so please correct me if I am wrong) the PCI dma mapping code is 
> abstracted by struct dma_mapping_ops. I.e. there are currently four 
> possible implementations for x86_64 (see
> linux-2.6/arch/x86_64/kernel/)
> 
> 1. pci-nommu.c : no IOMMU at all (e.g. because you have < 4 GB memory)
>    Kernel boot message: "PCI-DMA: Disabling IOMMU."
> 
> 2. pci-gart.c : (AMD) Hardware-IOMMU.
>    Kernel boot message: "PCI-DMA: using GART IOMMU" (this message
>    first appeared in 2.6.16)
> 
> 3. pci-swiotlb.c : Software-IOMMU (used e.g. if there is no hw iommu)
>    Kernel boot message: "PCI-DMA: Using software bounce buffering 
>    for IO (SWIOTLB)"

Used if there's no HW IOMMU *and* it's needed (because you have >4GB
memory) or you told the kernel to use it (iommu=soft).

> 4. pci-calgary.c : Calgary HW-IOMMU from IBM; used in pSeries servers. 
>    This HW-IOMMU supports dma address mapping with memory proctection,
>    etc.
>    Kernel boot message: "PCI-DMA: Using Calgary IOMMU" (since
>    2.6.18!)

Calgary is found in pSeries servers, but also in high-end xSeries
(Intel based) servers. It would be a little awkward if pSeries servers
(which are based on PowerPC processors) used code under arch/x86-64
:-)

> BTW: It would be really great if this area of the kernel would get some 
> more and better documentation. The information at 
> linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> read the code to get a *rough* idea what all the "iommu=" options 
> actually do and how they interact.

Patches happily accepted :-)

Cheers,
Muli

[PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

> On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
> 
> > BTW: It would be really great if this area of the kernel would get some 
> > more and better documentation. The information at 
> > linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> > read the code to get a *rough* idea what all the "iommu=" options 
> > actually do and how they interact.
> 
> Patches happily accepted :-)

Well, you asked for it. :-) So here's my little contribution. Please 
*double* *check*!

(BTW: I would like to know what "DAC" and "SAC" means in this context)

===

From: Karsten Weiss <K.Weiss@science-computing.de>

Patch summary:

- Better explanation of some of the iommu kernel parameter options.
- "32MB<<order" instead of "32MB^order".
- Mention the default "order".
- SWIOTLB config help text
- removed the duplication of the iommu kernel parameter documentation.
- mention Documentation/x86_64/boot-options.txt in 
  Documentation/kernel-parameters.txt
- list the four existing PCI DMA mapping implementations of arch x86_64

Signed-off-by: Karsten Weiss <knweiss@science-computing.de>

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original	2006-12-14 11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c	2006-12-14 12:14:48.176967312 +0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
-         [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-          Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-            but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft	 Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-   nodac    Forbid DMA >4GB
-   panic    Force panic when IOMMU overflows
-*/
+/*
+ * See <Documentation/x86_64/boot-options.txt> for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
 	iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original	2006-12-14 11:37:35.832142506 +0100
+++ linux-2.6.19/arch/x86_64/Kconfig	2006-12-14 11:47:24.346056710 +0100
@@ -431,8 +431,8 @@
 	  on systems with more than 3GB. This is usually needed for USB,
 	  sound, many IDE/SATA chipsets and some other devices.
 	  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
-	  based IOMMU and a software bounce buffer based IOMMU used on Intel
-	  systems and as fallback.
+	  based hardware IOMMU and a software bounce buffer based IOMMU used
+	  on Intel systems and as fallback.
 	  The code is only active when needed (enough memory and limited
 	  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
 	  too.
@@ -458,6 +458,11 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
 	bool
+	help
+	  Support for a software bounce buffer based IOMMU used on Intel
+	  systems which don't have a hardware IOMMU. Using this code
+	  PCI devices with 32bit memory access only are able to be
+	  used on systems with more than 3 GB.
 
 config X86_MCE
 	bool "Machine check support" if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original	2006-12-14 11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt	2006-12-14 12:10:24.028009890 +0100
@@ -180,35 +180,66 @@
   pci=lastbus=NUMBER	       Scan upto NUMBER busses, no matter what the mptable says.
   pci=noacpi		Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
+
+ Currently four x86_64 PCI DMA mapping implementations exist:
+
+   1. <arch/x86_64/kernel/pci-nommu.c>: use no hardware/software IOMMU at all
+      (e.g. because you have < 3 GB memory).
+      Kernel boot message: "PCI-DMA: Disabling IOMMU"
+
+   2. <arch/x86_64/kernel/pci-gart.c>: AMD GART based hardware IOMMU.
+      Kernel boot message: "PCI-DMA: using GART IOMMU"
+
+   3. <arch/x86_64/kernel/pci-swiotlb.c> : Software IOMMU implementation. Used
+      e.g. if there is no hardware IOMMU in the system and it is need because
+      you have >3GB memory or told the kernel to us it (iommu=soft))
+      Kernel boot message: "PCI-DMA: Using software bounce buffering
+      for IO (SWIOTLB)"
+
+   4. <arch/x86_64/pci-calgary.c> : IBM Calgary hardware IOMMU. Used in IBM
+      pSeries and xSeries servers. This hardware IOMMU supports DMA address
+      mapping with memory protection, etc.
+      Kernel boot message: "PCI-DMA: Using Calgary IOMMU" 
 
  iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
          [,forcesac][,fullflush][,nomerge][,noaperture]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do SG merging. Implies force (experimental)
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft	 Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-	    When off all DMA over >4GB is forced through an IOMMU or bounce
-	    buffering.
-   nodac    Forbid DMA >4GB
-   panic    Always panic when IOMMU overflows
+   size             set size of IOMMU (in bytes)
+   noagp            don't initialize the AGP driver and use full aperture.
+   off              don't initialize and use any kind of IOMMU.
+   leak             turn on simple iommu leak tracing (only when
+                    CONFIG_IOMMU_LEAK is on)
+   memaper[=order]  allocate an own aperture over RAM with size 32MB<<order.
+                    (default: order=1, i.e. 64MB)
+   noforce          don't force hardware IOMMU usage when it is not needed.
+                    (default).
+   force            Force the use of the hardware IOMMU even when it is
+                    not actually needed (e.g. because < 3 GB memory).
+   merge            Do scather-gather (SG) merging. Implies force (experimental)
+   nomerge          Don't do scather-gather (SG) merging.
+   forcesac         For SAC mode for masks <40bits  (experimental)
+   fullflush        Flush AMD GART based hardware IOMMU on each allocation
+                    (default)
+   nofullflush      Don't use IOMMU fullflush
+   allowed          overwrite iommu off workarounds for specific chipsets.
+   soft             Use software bounce buffering (SWIOTLB) (default for Intel
+                    machines). This can be used to prevent the usage
+                    of a available hardware IOMMU.
+   noaperture       Ask the AMD GART based hardware IOMMU driver not to 
+                    touch the aperture for AGP.
+   allowdac         Allow DMA >4GB
+                    When off all DMA over >4GB is forced through an IOMMU or
+                    bounce buffering.
+   nodac            Forbid DMA >4GB
+   panic            Always panic when IOMMU overflows
 
   swiotlb=pages[,force]
+   pages            Prereserve that many 128K pages for the software IO bounce
+                    buffering.
+   force            Force all IO through the software TLB.
 
-  pages  Prereserve that many 128K pages for the software IO bounce buffering.
-  force  Force all IO through the software TLB.
+  Settings for the IBM Calgary hardware IOMMU currently found in IBM
+  pSeries and xSeries machines:
 
   calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
   calgary=[translate_empty_slots]
--- linux-2.6.19/Documentation/kernel-parameters.txt.original	2006-12-14 11:03:46.584429749 +0100
+++ linux-2.6.19/Documentation/kernel-parameters.txt	2006-12-14 11:11:22.172025378 +0100
@@ -104,6 +104,9 @@
 Do not modify the syntax of boot loader parameters without extreme
 need or coordination with <Documentation/i386/boot.txt>.
 
+There are also arch-specific kernel-parameters not documented here.
+See for example <Documentation/x86_64/boot-options.txt>.
+
 Note that ALL kernel parameters listed below are CASE SENSITIVE, and that
 a trailing = on the name of any parameter states that that parameter will
 be entered as an environment variable, whereas its absence indicates that

-- 
__________________________________________creating IT solutions
Dipl.-Inf. Karsten Weiss               science + computing ag
phone:    +49 7071 9457 452            Hagellocher Weg 73
teamline: +49 7071 9457 681            72070 Tuebingen, Germany
email:    knweiss@science-computing.de www.science-computing.de

Re: [PATCH] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:38:08PM +0100, Karsten Weiss wrote:
> On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
> 
> > On Wed, Dec 13, 2006 at 09:34:16PM +0100, Karsten Weiss wrote:
> > 
> > > BTW: It would be really great if this area of the kernel would get some 
> > > more and better documentation. The information at 
> > > linux-2.6/Documentation/x86_64/boot_options.txt is very terse. I had to 
> > > read the code to get a *rough* idea what all the "iommu=" options 
> > > actually do and how they interact.
> > 
> > Patches happily accepted :-)
> 
> Well, you asked for it. :-) So here's my little contribution. Please 
> *double* *check*!

Looks good, some nits below.

> (BTW: I would like to know what "DAC" and "SAC" means in this
> context)

Single / Double Address Cycle. DAC is used with 32-bit PCI to push a
64-bit address in two cycles.

> @@ -458,6 +458,11 @@
>  # need this always selected by IOMMU for the VIA workaround
>  config SWIOTLB
>  	bool
> +	help
> +	  Support for a software bounce buffer based IOMMU used on Intel
> +	  systems which don't have a hardware IOMMU. Using this code
> +	  PCI devices with 32bit memory access only are able to be
> +	  used on systems with more than 3 GB.

I would rephrase as follows: "Support for software bounce buffers used
on x86-64 systems which don't have a hardware IOMMU. Using this PCI
devices which can only access 32-bits of memory can be used on systems
with more than 3 GB of memory".

> +   size             set size of IOMMU (in bytes)

Due to historical precedence, some of these options are only valid for
GART. Perhaps mention for each option which IOMMUs it is valid for or
split them on a per IOMMU basis?

This one (size) is gart only.

> +   noagp            don't initialize the AGP driver and use full
>     aperture.

gart only.

> +   off              don't initialize and use any kind of IOMMU.

all.

> +   leak             turn on simple iommu leak tracing (only when
> +                    CONFIG_IOMMU_LEAK is on)

gart only.

> +   memaper[=order]  allocate an own aperture over RAM with size 32MB<<order.
> +                    (default: order=1, i.e. 64MB)

gart only.

> +   noforce          don't force hardware IOMMU usage when it is not needed.
> +                    (default).

all.

> +   force            Force the use of the hardware IOMMU even when it is
> +                    not actually needed (e.g. because < 3 GB
>                      memory).

all.

> +   merge            Do scather-gather (SG) merging. Implies force
>     (experimental)

gart only.

> +   nomerge          Don't do scather-gather (SG) merging.

gart only.

> +   forcesac         For SAC mode for masks <40bits  (experimental)

gart only.

> +   fullflush        Flush AMD GART based hardware IOMMU on each allocation
> +                    (default)

gart only.

> +   nofullflush      Don't use IOMMU fullflush

gart only.

> +   allowed          overwrite iommu off workarounds for specific
>     chipsets.

gart only.

> +   soft             Use software bounce buffering (SWIOTLB) (default for Intel
> +                    machines). This can be used to prevent the usage
> +                    of a available hardware IOMMU.

all.

> +   noaperture       Ask the AMD GART based hardware IOMMU driver not to 
> +                    touch the aperture for AGP.

gart only.

> +   allowdac         Allow DMA >4GB
> +                    When off all DMA over >4GB is forced through an IOMMU or
> +                    bounce buffering.

gart only.

> +   nodac            Forbid DMA >4GB

gart only.

> +   panic            Always panic when IOMMU overflows

gart and Calgary.

The rest looks good. Please resend and I'll add my Acked-by.

Cheers,
Muli

[PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:

> The rest looks good. Please resend and I'll add my Acked-by.

Thanks a lot for your comments and suggestions. Here's my 2nd try:

===

From: Karsten Weiss <knweiss@science-computing.de>

$ diffstat ~/iommu-patch_v2.patch
 Documentation/kernel-parameters.txt   |    3
 Documentation/x86_64/boot-options.txt |  104 +++++++++++++++++++++++-----------
 arch/x86_64/Kconfig                   |   10 ++-
 arch/x86_64/kernel/pci-dma.c          |   28 +--------
 4 files changed, 87 insertions(+), 58 deletions(-)

Patch description:

- add SWIOTLB config help text
- mention Documentation/x86_64/boot-options.txt in
  Documentation/kernel-parameters.txt
- remove the duplication of the iommu kernel parameter documentation.
- Better explanation of some of the iommu kernel parameter options.
- "32MB<<order" instead of "32MB^order".
- Mention the default "order" value.
- list the four existing PCI-DMA mapping implementations of arch x86_64
- group the iommu= option keywords by PCI-DMA mapping implementation.
- Distinguish iommu= option keywords from number arguments.
- Explain the meaning of DAC and SAC.

Signed-off-by: Karsten Weiss <knweiss@science-computing.de>

---

--- linux-2.6.19/arch/x86_64/kernel/pci-dma.c.original	2006-12-14 11:15:38.348598021 +0100
+++ linux-2.6.19/arch/x86_64/kernel/pci-dma.c	2006-12-14 12:14:48.176967312 +0100
@@ -223,30 +223,10 @@
 }
 EXPORT_SYMBOL(dma_set_mask);
 
-/* iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
-         [,forcesac][,fullflush][,nomerge][,biomerge]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do lazy merging. This may improve performance on some block devices.
-          Implies force (experimental)
-   biomerge Do merging at the BIO layer. This is more efficient than merge,
-            but should be only done with very big IOMMUs. Implies merge,force.
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft	 Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-   nodac    Forbid DMA >4GB
-   panic    Force panic when IOMMU overflows
-*/
+/*
+ * See <Documentation/x86_64/boot-options.txt> for the iommu kernel parameter
+ * documentation.
+ */
 __init int iommu_setup(char *p)
 {
 	iommu_merge = 1;
--- linux-2.6.19/arch/x86_64/Kconfig.original	2006-12-14 11:37:35.832142506 +0100
+++ linux-2.6.19/arch/x86_64/Kconfig	2006-12-14 14:01:24.009193996 +0100
@@ -431,8 +431,8 @@
 	  on systems with more than 3GB. This is usually needed for USB,
 	  sound, many IDE/SATA chipsets and some other devices.
 	  Provides a driver for the AMD Athlon64/Opteron/Turion/Sempron GART
-	  based IOMMU and a software bounce buffer based IOMMU used on Intel
-	  systems and as fallback.
+	  based hardware IOMMU and a software bounce buffer based IOMMU used
+	  on Intel systems and as fallback.
 	  The code is only active when needed (enough memory and limited
 	  device) unless CONFIG_IOMMU_DEBUG or iommu=force is specified
 	  too.
@@ -458,6 +458,12 @@
 # need this always selected by IOMMU for the VIA workaround
 config SWIOTLB
 	bool
+	help
+	  Support for software bounce buffers used on x86-64 systems
+	  which don't have a hardware IOMMU (e.g. the current generation
+	  of Intel's x86-64 CPUs). Using this PCI devices which can only
+	  access 32-bits of memory can be used on systems with more than
+	  3 GB of memory. If unsure, say Y.
 
 config X86_MCE
 	bool "Machine check support" if EMBEDDED
--- linux-2.6.19/Documentation/x86_64/boot-options.txt.original	2006-12-14 11:11:32.099300994 +0100
+++ linux-2.6.19/Documentation/x86_64/boot-options.txt	2006-12-14 14:14:55.869560532 +0100
@@ -180,39 +180,79 @@
   pci=lastbus=NUMBER	       Scan upto NUMBER busses, no matter what the mptable says.
   pci=noacpi		Don't use ACPI to set up PCI interrupt routing.
 
-IOMMU
+IOMMU (input/output memory management unit)
 
- iommu=[size][,noagp][,off][,force][,noforce][,leak][,memaper[=order]][,merge]
-         [,forcesac][,fullflush][,nomerge][,noaperture]
-   size  set size of iommu (in bytes)
-   noagp don't initialize the AGP driver and use full aperture.
-   off   don't use the IOMMU
-   leak  turn on simple iommu leak tracing (only when CONFIG_IOMMU_LEAK is on)
-   memaper[=order] allocate an own aperture over RAM with size 32MB^order.
-   noforce don't force IOMMU usage. Default.
-   force  Force IOMMU.
-   merge  Do SG merging. Implies force (experimental)
-   nomerge Don't do SG merging.
-   forcesac For SAC mode for masks <40bits  (experimental)
-   fullflush Flush IOMMU on each allocation (default)
-   nofullflush Don't use IOMMU fullflush
-   allowed  overwrite iommu off workarounds for specific chipsets.
-   soft	 Use software bounce buffering (default for Intel machines)
-   noaperture Don't touch the aperture for AGP.
-   allowdac Allow DMA >4GB
-	    When off all DMA over >4GB is forced through an IOMMU or bounce
-	    buffering.
-   nodac    Forbid DMA >4GB
-   panic    Always panic when IOMMU overflows
-
-  swiotlb=pages[,force]
-
-  pages  Prereserve that many 128K pages for the software IO bounce buffering.
-  force  Force all IO through the software TLB.
-
-  calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
-  calgary=[translate_empty_slots]
-  calgary=[disable=<PCI bus number>]
+ Currently four x86-64 PCI-DMA mapping implementations exist:
+
+   1. <arch/x86_64/kernel/pci-nommu.c>: use no hardware/software IOMMU at all
+      (e.g. because you have < 3 GB memory).
+      Kernel boot message: "PCI-DMA: Disabling IOMMU"
+
+   2. <arch/x86_64/kernel/pci-gart.c>: AMD GART based hardware IOMMU.
+      Kernel boot message: "PCI-DMA: using GART IOMMU"
+
+   3. <arch/x86_64/kernel/pci-swiotlb.c> : Software IOMMU implementation. Used
+      e.g. if there is no hardware IOMMU in the system and it is need because
+      you have >3GB memory or told the kernel to us it (iommu=soft))
+      Kernel boot message: "PCI-DMA: Using software bounce buffering
+      for IO (SWIOTLB)"
+
+   4. <arch/x86_64/pci-calgary.c> : IBM Calgary hardware IOMMU. Used in IBM
+      pSeries and xSeries servers. This hardware IOMMU supports DMA address
+      mapping with memory protection, etc.
+      Kernel boot message: "PCI-DMA: Using Calgary IOMMU" 
+
+ iommu=[<size>][,noagp][,off][,force][,noforce][,leak[=<nr_of_leak_pages>]
+         [,memaper[=<order>]][,merge][,forcesac][,fullflush][,nomerge][,noaperture]
+
+  General iommu options:
+    off                Don't initialize and use any kind of IOMMU.
+    noforce            Don't force hardware IOMMU usage when it is not needed.
+                       (default).
+    force              Force the use of the hardware IOMMU even when it is
+                       not actually needed (e.g. because < 3 GB memory).
+    soft               Use software bounce buffering (SWIOTLB) (default for
+                       Intel machines). This can be used to prevent the usage
+                       of an available hardware IOMMU.
+
+  iommu options only relevant to the AMD GART hardware IOMMU:
+    <size>             Set the size of the remapping area in bytes.
+    allowed            Overwrite iommu off workarounds for specific chipsets.
+    fullflush          Flush IOMMU on each allocation (default).
+    nofullflush        Don't use IOMMU fullflush.
+    leak               Turn on simple iommu leak tracing (only when
+                       CONFIG_IOMMU_LEAK is on). Default number of leak pages
+                       is 20.
+    memaper[=<order>]  Allocate an own aperture over RAM with size 32MB<<order.
+                       (default: order=1, i.e. 64MB)
+    merge              Do scather-gather (SG) merging. Implies "force"
+                       (experimental).
+    nomerge            Don't do scather-gather (SG) merging.
+    noaperture         Ask the IOMMU not to touch the aperture for AGP.
+    forcesac           Force single-address cycle (SAC) mode for masks <40bits
+                       (experimental).
+    noagp              Don't initialize the AGP driver and use full aperture.
+    allowdac           Allow double-address cycle (DAC) mode, i.e. DMA >4GB.
+                       DAC is used with 32-bit PCI to push a 64-bit address in
+                       two cycles. When off all DMA over >4GB is forced through
+                       an IOMMU or software bounce buffering.
+    nodac              Forbid DAC mode, i.e. DMA >4GB.
+    panic              Always panic when IOMMU overflows.
+ 
+  iommu options only relevant to the software bounce buffering (SWIOTLB) IOMMU
+  implementation:
+    swiotlb=<pages>[,force]
+    <pages>            Prereserve that many 128K pages for the software IO
+                       bounce buffering.
+    force              Force all IO through the software TLB.
+
+  Settings for the IBM Calgary hardware IOMMU currently found in IBM
+  pSeries and xSeries machines:
+
+    calgary=[64k,128k,256k,512k,1M,2M,4M,8M]
+    calgary=[translate_empty_slots]
+    calgary=[disable=<PCI bus number>]
+    panic              Always panic when IOMMU overflows
 
     64k,...,8M - Set the size of each PCI slot's translation table
     when using the Calgary IOMMU. This is the size of the translation
--- linux-2.6.19/Documentation/kernel-parameters.txt.original	2006-12-14 11:03:46.584429749 +0100
+++ linux-2.6.19/Documentation/kernel-parameters.txt	2006-12-14 11:11:22.172025378 +0100
@@ -104,6 +104,9 @@
 Do not modify the syntax of boot loader parameters without extreme
 need or coordination with <Documentation/i386/boot.txt>.
 
+There are also arch-specific kernel-parameters not documented here.
+See for example <Documentation/x86_64/boot-options.txt>.
+
 Note that ALL kernel parameters listed below are CASE SENSITIVE, and that
 a trailing = on the name of any parameter states that that parameter will
 be entered as an environment variable, whereas its absence indicates that


-- 
__________________________________________creating IT solutions
Dipl.-Inf. Karsten Weiss               science + computing ag
phone:    +49 7071 9457 452            Hagellocher Weg 73
teamline: +49 7071 9457 681            72070 Tuebingen, Germany
email:    knweiss@science-computing.de www.science-computing.de

Re: [PATCH 2nd try] Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:16:31PM +0100, Karsten Weiss wrote:
> On Thu, 14 Dec 2006, Muli Ben-Yehuda wrote:
> 
> > The rest looks good. Please resend and I'll add my Acked-by.
> 
> Thanks a lot for your comments and suggestions. Here's my 2nd try:
> 
> ===
> 
> From: Karsten Weiss <knweiss@science-computing.de>
> 
> $ diffstat ~/iommu-patch_v2.patch
>  Documentation/kernel-parameters.txt   |    3
>  Documentation/x86_64/boot-options.txt |  104 +++++++++++++++++++++++-----------
>  arch/x86_64/Kconfig                   |   10 ++-
>  arch/x86_64/kernel/pci-dma.c          |   28 +--------
>  4 files changed, 87 insertions(+), 58 deletions(-)
> 
> Patch description:
> 
> - add SWIOTLB config help text
> - mention Documentation/x86_64/boot-options.txt in
>   Documentation/kernel-parameters.txt
> - remove the duplication of the iommu kernel parameter documentation.
> - Better explanation of some of the iommu kernel parameter options.
> - "32MB<<order" instead of "32MB^order".
> - Mention the default "order" value.
> - list the four existing PCI-DMA mapping implementations of arch x86_64
> - group the iommu= option keywords by PCI-DMA mapping implementation.
> - Distinguish iommu= option keywords from number arguments.
> - Explain the meaning of DAC and SAC.
> 
> Signed-off-by: Karsten Weiss <knweiss@science-computing.de>

Acked-by: Muli Ben-Yehuda <muli@il.ibm.com>

Cheers,
Muli

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 14493 bytes --]

Hi my friends....

It became a little bit silent about this issue... any new ideas or results?



Karsten Weiss wrote:
> BTW: Did someone already open an official bug at
> http://bugzilla.kernel.org ?
Karsten, did you already file a bug?



I told the whole issue to the Debian people which are about to release
etch and suggested them to use iommu=soft by default.
This brings me to:
Chris Wedgwood wrote:
> Does anyone have an amd64 with an nforce4 chipset and >4GB that does
> NOT have this problem? If so it might be worth chasing the BIOS
> vendors to see what errata they are dealing with.
John Chaves replied and claimed that he wouldn't suffer from that
problem (I've CC'ed him to this post).
You can read his message at the bottom of this post.
@ John: Could you please tell us in detail how you've tested your system?



Muli told us some information about the iommu options (when he
discuessed Karstens patch) has anybody made tests with the other iommu
options?



Ok and what does it all come down to? We still don't know the exact
reason...
Perhaps a kernel bug, a Opteron and/or Chipset bug,.. and perhaps there
are even some BIOSes that solve the issue...

For the kernel-bug reason,... who is the responsible developer for the
relevant code? Can we contact him to read our threads and perhaps review
the code?

Is anyone able (or wants to try) to inform AMD and/or Nvidia about the
issue (perhaps with pointing to that thread).

Someone might even try to contact some board vendors (some of us seem to
have Tyan boards). Although I'm in contact with the German support Team
of Tyan, I wasn't very successful with the US team... perhaps they have
other ideas.

Last but not least.... if we don't find a solution what should we do?
In my opinion at least the following:
1) Inform other OS communities (*BSD) and point the to our thread. Some
of you claimed that Windows wouldn't use the hwiommu at all so I think
we don't have to contact big evil.
2) Contact the major Linux Distributions (I've already did it for
Debian) and inform them about the potential issue and pointing them to
this thread (where one can find all the relevant information, I think)
3) Workaround for the kernel:
I have to less knowledge to know exactly what to do but I remember there
are other fixes for mainboard flaws and buggy chipsets in the kernel
(e.g. the RZ1000 or something like this in the "old" IDE driver)...
Perhaps someone (who knows what to do ;-) ) could write some code that
automatically uses iommu=soft,... but then we have the question: In
which case :-( . I imagine that the AMD users who don't suffer from this
issue would like to continue using their hwiommus..


What I'm currently plan to do:
1) If know one else is willing to try contacting AMD/Nvidia,.. I'd try
again.
2) I told you that I'm going to test the whole issue in the Leibniz
Supercomputing Centre where I work as student...
This is a little bit delayed (organisational problems :-) )
Anyway,... I'm not only going to test it on our Linux Cluster but also
some Sun Fire's (whe have maaaannnnny of them ;-) ). According to my
boss they have nvidia chipsets... (He is probably contacting Sun for the
issue).



So much for now.

Best wishes,
Chris.


John Chaves message:
Here's another data point in case it helps.
The following system does *not* have the data corruption issue.

Motherboard: Iwill DK88 <http://www.iwill.net/product_2.asp?p_id=102>
Chipset: NVIDIA nForce4 Professional 2200
CPUs: Two Dual Core AMD Opteron(tm) Processor 280
Memory: 32GB
Disks: Four 500GB SATA in linux RAID1 over RAID0 setup
Kernel: 2.6.18

This system is a workhorse with extreme disk I/O of huge files,
and the nature of the work done would have revealed data
corruption pretty quickly.

FWIW,
John Chaves

His lspic:
0000:00:00.0 Memory controller: nVidia Corporation CK804 Memory
Controller (rev a3)
Flags: bus master, 66MHz, fast devsel, latency 0
Capabilities: [44] #08 [01e0]
Capabilities: [e0] #08 [a801]

0000:00:01.0 ISA bridge: nVidia Corporation CK804 ISA Bridge (rev a3)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0

0000:00:01.1 SMBus: nVidia Corporation CK804 SMBus (rev a2)
Subsystem: nVidia Corporation: Unknown device cb84
Flags: 66MHz, fast devsel, IRQ 9
I/O ports at d400 [size=32]
I/O ports at 4c00 [size=64]
I/O ports at 4c40 [size=64]
Capabilities: [44] Power Management version 2

0000:00:02.0 USB Controller: nVidia Corporation CK804 USB Controller
(rev a2) (prog-if 10 [OHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 209
Memory at feafc000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2

0000:00:02.1 USB Controller: nVidia Corporation CK804 USB Controller
(rev a3) (prog-if 20 [EHCI])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 193
Memory at feafdc00 (32-bit, non-prefetchable) [size=256]
Capabilities: [44] #0a [2098]
Capabilities: [80] Power Management version 2

0000:00:06.0 IDE interface: nVidia Corporation CK804 IDE (rev a2)
(prog-if 8a [Master SecP PriP])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0
I/O ports at 3000 [size=16]
Capabilities: [44] Power Management version 2

0000:00:07.0 IDE interface: nVidia Corporation CK804 Serial ATA
Controller (rev a3) (prog-if 85 [Master SecO PriO])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 193
I/O ports at e800 [size=8]
I/O ports at e400 [size=4]
I/O ports at e000 [size=8]
I/O ports at dc00 [size=4]
I/O ports at d800 [size=16]
Memory at feafe000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2

0000:00:08.0 IDE interface: nVidia Corporation CK804 Serial ATA
Controller (rev a3) (prog-if 85 [Master SecO PriO])
Subsystem: nVidia Corporation: Unknown device cb84
Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 201
I/O ports at fc00 [size=8]
I/O ports at f800 [size=4]
I/O ports at f400 [size=8]
I/O ports at f000 [size=4]
I/O ports at ec00 [size=16]
Memory at feaff000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [44] Power Management version 2

0000:00:09.0 PCI bridge: nVidia Corporation CK804 PCI Bridge (rev a2)
(prog-if 01 [Subtractive decode])
Flags: bus master, 66MHz, fast devsel, latency 0
Bus: primary=00, secondary=05, subordinate=05, sec-latency=128
I/O behind bridge: 0000b000-0000bfff
Memory behind bridge: fc900000-fe9fffff
Prefetchable memory behind bridge: e0000000-e00fffff

0000:00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
Memory behind bridge: fc800000-fc8fffff
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Capabilities: [58] #08 [a800]
Capabilities: [80] #10 [0141]

0000:00:0c.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Memory behind bridge: fc700000-fc7fffff
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Capabilities: [58] #08 [a800]
Capabilities: [80] #10 [0141]

0000:00:0d.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Capabilities: [58] #08 [a800]
Capabilities: [80] #10 [0141]

0000:00:0e.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Capabilities: [40] Power Management version 2
Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
Capabilities: [58] #08 [a800]
Capabilities: [80] #10 [0141]

0000:00:18.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
Flags: fast devsel
Capabilities: [80] #08 [2101]
Capabilities: [a0] #08 [2101]
Capabilities: [c0] #08 [2101]

0000:00:18.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
Flags: fast devsel

0000:00:18.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
Flags: fast devsel

0000:00:18.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
Flags: fast devsel

0000:00:19.0 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] HyperTransport Technology Configuration
Flags: fast devsel
Capabilities: [80] #08 [2101]
Capabilities: [a0] #08 [2101]
Capabilities: [c0] #08 [2101]

0000:00:19.1 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Address Map
Flags: fast devsel

0000:00:19.2 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] DRAM Controller
Flags: fast devsel

0000:00:19.3 Host bridge: Advanced Micro Devices [AMD] K8
[Athlon64/Opteron] Miscellaneous Control
Flags: fast devsel

0000:03:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 11)
Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI
Express
Flags: bus master, fast devsel, latency 0, IRQ 185
Memory at fc7f0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Capabilities: [d0] #10 [0001]

0000:04:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721
Gigabit Ethernet PCI Express (rev 11)
Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI
Express
Flags: bus master, fast devsel, latency 0, IRQ 177
Memory at fc8f0000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/3 Enable-
Capabilities: [d0] #10 [0001]

0000:05:07.0 VGA compatible controller: ATI Technologies Inc Rage XL
(rev 27) (prog-if 00 [VGA])
Subsystem: ATI Technologies Inc Rage XL
Flags: bus master, stepping, medium devsel, latency 64, IRQ 10
Memory at fd000000 (32-bit, non-prefetchable) [size=16M]
I/O ports at b800 [size=256]
Memory at fe8ff000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at e0000000 [disabled] [size=128K]
Capabilities: [5c] Power Management version 2

0000:06:01.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8132 PCI-X
Bridge (rev 11) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 64
Bus: primary=06, secondary=08, subordinate=08, sec-latency=64
Capabilities: [60] Capabilities: [b8] #08 [8000]
Capabilities: [c0] #08 [0041]
Capabilities: [f4] #08 [a800]

0000:06:01.1 PIC: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
(rev 11) (prog-if 10 [IO-APIC])
Subsystem: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
Flags: bus master, medium devsel, latency 0
Memory at febfe000 (64-bit, non-prefetchable) [size=4K]

0000:06:02.0 PCI bridge: Advanced Micro Devices [AMD] AMD-8132 PCI-X
Bridge (rev 11) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 64
Bus: primary=06, secondary=07, subordinate=07, sec-latency=64
Capabilities: [60] Capabilities: [b8] #08 [8000]
Capabilities: [c0] #08 [8840]
Capabilities: [f4] #08 [a800]

0000:06:02.1 PIC: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
(rev 11) (prog-if 10 [IO-APIC])
Subsystem: Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
Flags: bus master, medium devsel, latency 0
Memory at febff000 (64-bit, non-prefetchable) [size=4K]

His /proc/cpuinfo:processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2400.020
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 4802.02
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 1
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2400.020
cache size : 1024 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 4799.29
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 2
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2400.020
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 4799.36
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp

processor : 3
vendor_id : AuthenticAMD
cpu family : 15
model : 33
model name : Dual Core AMD Opteron(tm) Processor 280
stepping : 2
cpu MHz : 2400.020
cache size : 1024 KB
physical id : 1
siblings : 2
core id : 1
cpu cores : 2
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt lm
3dnowext 3dnow pni lahf_lm cmp_legacy
bogomips : 4799.37
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp


[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[John A Chaves]

On Friday 22 December 2006 20:04, Christoph Anton Mitterer wrote:
> This brings me to:
> Chris Wedgwood wrote:
> > Does anyone have an amd64 with an nforce4 chipset and >4GB that does
> > NOT have this problem? If so it might be worth chasing the BIOS
> > vendors to see what errata they are dealing with.
> John Chaves replied and claimed that he wouldn't suffer from that
> problem (I've CC'ed him to this post).
> You can read his message at the bottom of this post.
> @ John: Could you please tell us in detail how you've tested your system?

I didn't need to run a specific test for this.  The normal workload of the
machine approximates a continuous selftest for almost the last year.

Large files (4-12GB is typical) are being continuously packed and unpacked
with gzip and bzip2.  Statistical analysis of the datasets is followed by
verification of the data, sometimes using diff, or md5sum, or python
scripts using numarray to mmap 2GB chunks at a time.  The machine
often goes for days with a load level of 20+ and 32GB RAM + another 32GB
swap in use.  It would be very unlikely for data corruption to go unnoticed.

When I first got the machine I did have some problems with disks being
dropped from the RAID and occasional log messages implicating the IOMMU.
But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
problem.

John

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

John A Chaves wrote:
> I didn't need to run a specific test for this.  The normal workload of the
> machine approximates a continuous selftest for almost the last year.
>
> Large files (4-12GB is typical) are being continuously packed and unpacked
> with gzip and bzip2.  Statistical analysis of the datasets is followed by
> verification of the data, sometimes using diff, or md5sum, or python
> scripts using numarray to mmap 2GB chunks at a time.  The machine
> often goes for days with a load level of 20+ and 32GB RAM + another 32GB
> swap in use.  It would be very unlikely for data corruption to go unnoticed.
>
> When I first got the machine I did have some problems with disks being
> dropped from the RAID and occasional log messages implicating the IOMMU.
> But that was with kernel 2.6.16.?, Kernels since 2.6.17 haven't had any
> problem.
>   
Ah thanks for that info,.. as far as I can tell,.. this "testing
environment" should have found any corruptions I there had been any.

So I think we could take this as our first working system where the
issue don't occur although we would expect it...

Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 107 bytes --]

Ah and I forgot,...

Did anyone made any test under Windows? I cannot set there iommu=soft,
can I?

Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Chris Wedgwood]

On Wed, Dec 13, 2006 at 08:20:59PM +0100, Christoph Anton Mitterer wrote:

> Did anyone made any test under Windows? I cannot set there
> iommu=soft, can I?

Windows never uses the hardware iommu, so it's always doing the
equivalent on iommu=soft

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

Chris Wedgwood wrote:
>> Did anyone made any test under Windows? I cannot set there
>> iommu=soft, can I?
>>     
> Windows never uses the hardware iommu, so it's always doing the
> equivalent on iommu=soft
>   
That would mean that I'm not able to reproduce the issue unter windows,
right?
Does that apply for all versions (up to and including Vista).

Don't understand me wrong,.. I don't use Windows (expect for upgrading
my Plextor firmware and EAC ;) )... but I ask because the more
information we get (even if it's not Linux specific) the more steps we
can take ;)

Chris.


[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Lennart Sorensen]

On Wed, Dec 13, 2006 at 08:57:23PM +0100, Christoph Anton Mitterer wrote:
> Don't understand me wrong,.. I don't use Windows (expect for upgrading
> my Plextor firmware and EAC ;) )... but I ask because the more
> information we get (even if it's not Linux specific) the more steps we
> can take ;)

I upgrade my plextor firmware using linux.  pxupdate for most devices,
and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
is one of the reasons I buy plextors.

--
Len Sorensen

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 486 bytes --]

Lennart Sorensen wrote:
> I upgrade my plextor firmware using linux.  pxupdate for most devices,
> and pxfw for new drivers (like the PX760).  Works perfectly for me.  It
> is one of the reasons I buy plextors.
Yes I know about it,.. although never tested it,... anyway the main
reason for Windows is Exact Audio Copy (but Andre Wiehthoff is working
on a C port :-D )

Unfortunately my PX760 seems to be defect,.. posted about the issue to
lkml but no success :-(

Best wishes,
Chris.


[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> Last week we did some more testing with the following result:
> 
> We could not reproduce the data corruption anymore if we boot the machines 
> with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> buffering instead of the hw-iommu. (As mentioned before, booting with 
> mem=2g works fine, too, because this disables the iommu altogether.)
> 
> I.e. on these systems the data corruption only happens if the hw-iommu 
> (PCI-GART) of the Opteron CPUs is in use.
> 
> Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> confirm this workaround, too.

What did you set the BIOS to when testing this setting?
Memory Hole enabled?  IOMMU enabled?

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Karsten Weiss]

On Wed, 13 Dec 2006, Erik Andersen wrote:

> On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> > Last week we did some more testing with the following result:
> > 
> > We could not reproduce the data corruption anymore if we boot the machines 
> > with the kernel parameter "iommu=soft" i.e. if we use software bounce 
> > buffering instead of the hw-iommu. (As mentioned before, booting with 
> > mem=2g works fine, too, because this disables the iommu altogether.)
> > 
> > I.e. on these systems the data corruption only happens if the hw-iommu 
> > (PCI-GART) of the Opteron CPUs is in use.
> > 
> > Christoph, Erik, Chris: I would appreciate if you would test and hopefully 
> > confirm this workaround, too.
> 
> What did you set the BIOS to when testing this setting?
> Memory Hole enabled?  IOMMU enabled?

"Memory hole mapping" was set to "hardware". With "disabled" we only
see 3 of our 4 GB memory.

Best regards,
Karsten

-- 
__________________________________________creating IT solutions
Dipl.-Inf. Karsten Weiss               science + computing ag
phone:    +49 7071 9457 452            Hagellocher Weg 73
teamline: +49 7071 9457 681            72070 Tuebingen, Germany
email:    knweiss@science-computing.de www.science-computing.de

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 272 bytes --]

Karsten Weiss wrote:
> "Memory hole mapping" was set to "hardware". With "disabled" we only
> see 3 of our 4 GB memory.
>   
That sounds reasonable,... I even only see 2,5 GB,.. as my memhole takes
1536 MB (don't ask me which PCI device needs that much address space ;) )

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> We could not reproduce the data corruption anymore if we boot
> the machines with the kernel parameter "iommu=soft" i.e. if we
> use software bounce buffering instead of the hw-iommu.

I just realized that booting with "iommu=soft" makes my pcHDTV
HD5500 DVB cards not work.  Time to go back to disabling the
memhole and losing 1 GB.  :-(

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

Erik Andersen wrote:
> I just realized that booting with "iommu=soft" makes my pcHDTV
> HD5500 DVB cards not work.  Time to go back to disabling the
> memhole and losing 1 GB.  :-(
Crazy,...
I have a Hauppauge Nova-T 500 DualDVB-T card,... I'll check it later if
I have the same problem and will inform you (please remember me if I
forget ;) )


Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Hi.

I've just looked for some kernel config options that might relate to our
issue:


1)
Old style AMD Opteron NUMA detection (CONFIG_K8_NUMA)
Enable K8 NUMA node topology detection.  You should say Y here if you
have a multi processor AMD K8 system. This uses an old method to read
the NUMA configuration directly from the builtin Northbridge of Opteron.
It is recommended to use X86_64_ACPI_NUMA instead, which also takes
priority if both are compiled in.

ACPI NUMA detection (CONFIG_X86_64_ACPI_NUMA)
Enable ACPI SRAT based node topology detection.

What should one select for the Opterons? And is it possible that this
has something to do with our datacorruption error?


2)
The same two questions for the memory model (Discontiguous or Sparse)



3)
The same two questions for CONFIG_MIGRATION ()


4)
And does someone know if the nforce/opteron iommu requires IBM Calgary
IOMMU support?



This is unrelated to our issue,.. but it would be nice if some of your
could send me their .config,.. I'd like to compare them with my own and
see if I could something tweak or so.
(Of course only people with 2x DualCore Systems ;) )



Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 12:33:23AM +0100, Christoph Anton Mitterer wrote:

> 4)
> And does someone know if the nforce/opteron iommu requires IBM Calgary
> IOMMU support?

It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
chipsets (AFAIK). However, compiling Calgary in should make no
difference, as we detect in run-time which IOMMU is found and the
machine.

Cheers,
Muli

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 753 bytes --]

Muli Ben-Yehuda wrote:
>> 4)
>> And does someone know if the nforce/opteron iommu requires IBM Calgary
>> IOMMU support?
>>     
> It doesn't, Calgary isn't found in machine with Opteron CPUs or NForce
> chipsets (AFAIK). However, compiling Calgary in should make no
> difference, as we detect in run-time which IOMMU is found and the
> machine.
Yes,.. I've read the relevant section shortly after sending that email ;-)

btw & for everybody:
I'm working (as student) at the LRZ (Leibniz Computing Centre) in Munich
where we have very large Linux Cluster and lots of different other
machines,...
I'm going to test for that error on most of the different types of
systems we have,.. and will inform you about my results (if they're
interesting).

Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Wed, Dec 13, 2006 at 01:29:25PM -0700, Erik Andersen wrote:
> On Mon Dec 11, 2006 at 10:24:02AM +0100, Karsten Weiss wrote:
> > We could not reproduce the data corruption anymore if we boot
> > the machines with the kernel parameter "iommu=soft" i.e. if we
> > use software bounce buffering instead of the hw-iommu.
> 
> I just realized that booting with "iommu=soft" makes my pcHDTV
> HD5500 DVB cards not work.  Time to go back to disabling the
> memhole and losing 1 GB.  :-(

That points to a bug in the driver (likely) or swiotlb (unlikely), as
the IOMMU in use should be transparent to the driver. Which driver is
it?

Cheers,
Muli

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Erik Andersen]

On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
> > I just realized that booting with "iommu=soft" makes my pcHDTV
> > HD5500 DVB cards not work.  Time to go back to disabling the
> > memhole and losing 1 GB.  :-(
> 
> That points to a bug in the driver (likely) or swiotlb (unlikely), as
> the IOMMU in use should be transparent to the driver. Which driver is
> it?

presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
to take an amazing number of drivers to make these devices
actually work...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Muli Ben-Yehuda]

On Thu, Dec 14, 2006 at 02:52:35AM -0700, Erik Andersen wrote:
> On Thu Dec 14, 2006 at 11:23:11AM +0200, Muli Ben-Yehuda wrote:
> > > I just realized that booting with "iommu=soft" makes my pcHDTV
> > > HD5500 DVB cards not work.  Time to go back to disabling the
> > > memhole and losing 1 GB.  :-(
> > 
> > That points to a bug in the driver (likely) or swiotlb (unlikely), as
> > the IOMMU in use should be transparent to the driver. Which driver is
> > it?
> 
> presumably one of cx88xx, cx88_blackbird, cx8800, cx88_dvb,
> cx8802, cx88_alsa, lgdt330x, tuner, cx2341x, btcx_risc,
> video_buf, video_buf_dvb, tveeprom, or dvb_pll.  It seems
> to take an amazing number of drivers to make these devices
> actually work...

Yikes! do you know which one actually handles the DMA mappings? I
suspect a missnig unmap or sync, which swiotlb requires to sync back
the bounce buffer with the driver's buffer.

Cheers,
Muli

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 9043 bytes --]

Hi everybody.

After my last mails to this issue (btw: anything new in the meantime? I
received no replys..) I wrote again to nvidia and AMD...
This time with some more success.

Below is the answer from Mr. Friedman to my mail. He says that he wasn't
able to reproduce the problem and asks for a testing system.
Unfortunately I cannot ship my system as this is my only home PC and I
need it for daily work. But perhaps someone else here might has a system
(with the error) that he can send to Nvidia...

I cc'ed Mr. Friedman so he'll read your replies.

To Mr. Friedman: What system did you exactly use for your testing?
(Hardware configuration, BIOS settings and so on). As we've seen before
it might be possible that some BIOSes correct the problem.

Best wishes,
Chris.




Lonni J Friedman wrote:
> Christoph,
> Thanks for your email.  I'm aware of the LKML threads, and have spent 
> considerable time attempting to reproduce this problem on one of our 
> reference motherboards without success.  If you could ship a system 
> which reliably reproduces the problem, I'd be happy to investigate further.
>
> Thanks,
> Lonni J Friedman
> NVIDIA Corporation
>
> Christoph Anton Mitterer wrote:
>   
>> Hi.
>>
>> First of all: This is only a copy from a thread to nvnews.net
>> (http://www.nvnews.net/vbulletin/showthread.php?t=82909). You probably
>> should read the description there.
>>
>> Please note that his is also a very important issue. It is most likely
>> not only Linux related but a general nforce chipset design flaw, so
>> perhaps you should forwad this mail to your engineers too. (Please CC me
>> in all mails).
>>
>> Also note: I'm not one of the normal "end users" with simple problems or
>> damaged hardware. I study computer science and work in one of Europes
>> largest supercomputing centres (Leibniz supercomputing centre).
>> Believe me: I know what I'm talking about.... and I'm investigating in
>> this issue (with many others) for some weeks now.
>>
>> Please answer either to the specific lkml thread, to the nvnews.net post
>> or directly to me (via email).
>> And I'd be grateful if you could give me email-addresses from your
>> developers or enginers, or even better, forward this email to them and
>> CC me. Of course I'll keep their emails-addresses absolutely confident
>> if you wish.
>>
>> Best wishes,
>> Christoph Anton Mitterer.
>> Munich University of Applied Sciences / Department of Mathematics and
>> Computer Science
>> Leibniz Supercomputing Centre / Department for High Performance
>> Computing and Compute Servers
>>
>>
>>
>>
>> Here is the copy:
>> Hi.
>>
>> I've already tried to "resolve" this via the nvidia knowledgebase but
>> either they don't want to know about that issue or there is noone who is
>> competent enought to give information/solutions about it.
>> They finally pointed me to this fourm and told me that Linux
>> <http://www.nvnews.net/vbulletin/showthread.php?t=82909#> support would
>> be handled here (they did not realise that this is probably a hardware
>> <http://www.nvnews.net/vbulletin/showthread.php?t=82909#> flaw and not
>> OS related).
>>
>> I must admit that I'm a little bit bored with Nvidia's policy in such
>> matters and thus I only describe the problem in brief.
>> If here is any competent chipset engineer who reads this, than he might
>> read the main discussion-thread (and some spin-off threads) of the issue
>> which takes place at the linux-kernel mailing list (again this is
>> probably not Linux related).
>> You can find the archive here:
>> http://marc.theaimsgroup.com/?t=116502121800001&r=1&w=2
>> <http://marc.theaimsgroup.com/?t=116502121800001&r=1&w=2>
>>
>>
>> Now a short description:
>> -I (and many others) found a data corruption issue that happens on AMD
>> Opteron / Nvidia chipset systems
>> <http://www.nvnews.net/vbulletin/showthread.php?t=82909#>.
>>
>> -What happens: If one reads/writes large amounts of data there are errors.
>> We test this the following way: Create some test data (huge amounts
>> of),.. make md5sums of it (or with other hash algorithms), then verify
>> them over and over.
>> The test shoes differences (refer the lkml thread for more information
>> about this). Always at differnt files (!!!!). It may happen at read AND
>> write access <http://www.nvnews.net/vbulletin/showthread.php?t=82909#>.
>> Note that even for affected users the error occurs rarely (but this is
>> of course still far to often): My personal tests shows about the following:
>> Test data: 30GB (of random data), I verify sha512sum 50 times (that is
>> what I call one complete test). So I verify 30*50GB. In one complete
>> test there are about 1-3 files with differences. With about 100
>> corrupted bytes (at leas very low data sizes, far below an MB)
>>
>> -It probably happens with all the nforce chipsets (see the lkml thread
>> where everybody tells his hardware)
>>
>> -The reasons are not single hardware defects (dozens of hight quality
>> memory <http://www.nvnews.net/vbulletin/showthread.php?t=82909#>, CPU,
>> PCI bus, HDD bad block scans, PCI parity, ECC, etc. tests showed this,
>> and even with different hardware compontents the issue remained)
>>
>> -It is probably not an Operating System related bug, although Windows
>> won't suffer from it. The reason therefore is, that windows is (too
>> stupid) ... I mean unable to use the hardware iommu at all.
>>
>> -It happens with both, PATA and SATA disk. To be exact: It is may that
>> this has nothing special to do with harddisks at all.
>> It is probably PCI-DMA related (see lkml for more infos and reasons for
>> this thesis).
>>
>> -Only users with much main memory (don't know the exact value by hard
>> and I'm to lazy to look it up)... say 4GB will suffer from this problem.
>> Why? Only users who need the memory hole mapping and the iommu will
>> suffer from the problem (this is why we think it is chipset related).
>>
>> -We found two "workarounds" but these have both big problems:
>> Workaround 1: Disable Memory Hole Mapping in the system BIOS at all.
>> The issue no longer occurs, BUT you loose a big part of your main memory
>> (depending on the size of the memhole, which itself depends on the PCI
>> devices). In my case I loose 1,5GB from my 4GB. Most users will probably
>> loose 1GB.
>> => inacceptable
>>
>> Workaround 2: As told Windows won't suffer from the problem because it
>> always uses an software iommu. (btw: the same applies for Intel CPUs
>> with EMT64/Intel 64,.. these CPUs don't even have a hardware iommu).
>> Linux is able to use the hardware iommu (which of course accelerates the
>> whole system).
>> If you tell the kernel (Linux) to use a software iommu (with the kernel
>> parameter iommu=soft),.. the issue won't appear.
>> => this is better than workaround 1 but still not really acceptable.
>> Why? There are some following problems:
>>
>> The hardware iommu and systems with such big main memory is largely used
>> in computing centres. Those groups won't abdicate the hwiommu in
>> general, simply because some Opteron (and perhaps Athlon) / Nvidia
>> combinations make problems.
>> (I can tell this because I work at the Leibniz Supercomputing Centre,..
>> one of the largest in Europe)
>>
>> But as we don't know the exact reason for the issue, we cannot
>> selectively switch the iommu=soft for affected
>> mainboards/chipsets/cpu-steppings/and alike.
>>
>> We'd have to use a kernel wide iommu=soft as a catchall solution.
>> But it is highly unlikely that this is accepted by the Linux community
>> (not to talk about end users like the supercomputing centres) and I
>> don't want to talk about other OS'es.
>>
>>
>> So we (and of course all, and especially professional, customers) need
>> Nvidias help.
>>
>> Perhaps this might be solvable via BIOS fixes, but of course not by the
>> stupid-solution "disable hwiommu via the BIOS".
>> Perhaps the reason is a Linux kernel bug (although this is highly unlikely).
>> Last but not least,.. perhaps this is AMD Opteron/Athlon (Note: These
>> CPUs have the memory controllers directly integrated) issue and/or
>> Nvidia nforce chipset issue.
>>
>> Regards,
>> Chris.
>> *
>> btw: For answers from Nvidia engineers/developers or end-users who
>> suffer from that issue too,... please post it to the lkml thread (see
>> above for the link) and if not possible here.
>> You may even contact me via email (calestyo@scientia.net) or personal
>> messages.*
>>
>> PS: Please post any other resources/links to threads about this or
>> similar problems.
>>     
>
> -----------------------------------------------------------------------------------
> This email message is for the sole use of the intended recipient(s) and may contain
> confidential information.  Any unauthorized review, use, disclosure or distribution
> is prohibited.  If you are not the intended recipient, please contact the sender by
> reply email and destroy all copies of the original message.
> -----------------------------------------------------------------------------------
>
>   


[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: data corruption with nvidia chipsets and IDE/SATA drives // memory hole mapping related bug?!

[Christoph Anton Mitterer]

[-- Attachment #1: Type: text/plain, Size: 136 bytes --]

Hi.

Just for you information: I've put the issue into the kernel.org bugzilla.
http://bugzilla.kernel.org/show_bug.cgi?id=7768

Chris.

[-- Attachment #2: calestyo.vcf --]
[-- Type: text/x-vcard, Size: 156 bytes --]

begin:vcard
fn:Mitterer, Christoph Anton
n:Mitterer;Christoph Anton
email;internet:calestyo@scientia.net
x-mozilla-html:TRUE
version:2.1
end:vcard