IDE DMA on AXP & barriers

IDE DMA on AXP & barriers

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 1892 bytes --]

Hi,

trying to find out, why the CMD646 (rev 01) IDE controller does corrupt my
file system on my PWS (Alpha) when writing in DMA mode, I started to wonder
about a number of things.

The pattern of the corruption is characteristic:
root@pws:/media/test/Dio/Diamonds # cmp -l copy original
  8193 377  70
  8194 373  71
  8195 220  40
  8196 144  50
 16385  70  47
 16386  71 367
 16387  40 152
 16388  50 341
 24577  47  20
 24578 367 342
 24579 152 136
 24580 341 202

always the first 4 bytes of a page still showing the old contents. 
To me this looks like either a some very weird bug in the chipset or more
likely like races between BM DMA and the CPU write buffers/caches. The
latter can easily be the case on Alpha, as there is _no_ implicit ordering
on any operation on the Alpha, not even interrupts, PAL calls, ...

Following the latter somewhat more, I found that the pcilogicisp driver
which works well, does have a number of mb() commands; the whole IDE code
does not have any of those.
I would imagine that the following barriers are nacessary:
* After setting up the IDE DMA tables (PRD) and having the data there,
  we need to have a barrier before telling the controller to do DMA.
* After the controller is done with it, we need to make sure the
  data is in mem before we use it (i.e. we need some mb() equiv on the
  controller)
* Some more less obvious ones probably ...

The Alpha Architecture manual sec. 5.6.4 gives some more info on how to
insure ordering of concurrent data access on shared memory.

Anyone familiar with IDE able to identify the necessary places in ide-dma
and/or cmd64x ?

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, DE                                SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: IDE DMA on AXP & barriers

[Ivan Kokshaysky]

On Thu, Dec 06, 2001 at 06:13:15AM +0100, Kurt Garloff wrote:
> I would imagine that the following barriers are nacessary:
> * After setting up the IDE DMA tables (PRD) and having the data there,
>   we need to have a barrier before telling the controller to do DMA.

Actually, we have more than one - the memory barriers are hidden in outX()
macros.

> * After the controller is done with it, we need to make sure the
>   data is in mem before we use it (i.e. we need some mb() equiv on the
>   controller)

Sure, otherwise the controller just wouldn't work, and not only on alpha.
I never had any problems with IDE DMA on lx164 (which has exactly the same
IDE chip), but it was back in 2.3 or 2.4-test times...
Maybe something got broken since then - I could check this (if I find
a spare IDE drive).

Ivan.

Re: IDE DMA on AXP & barriers

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

On Thu, Dec 06, 2001 at 12:59:35PM +0300, Ivan Kokshaysky wrote:
> On Thu, Dec 06, 2001 at 06:13:15AM +0100, Kurt Garloff wrote:
> > I would imagine that the following barriers are nacessary:
> > * After setting up the IDE DMA tables (PRD) and having the data there,
> >   we need to have a barrier before telling the controller to do DMA.
> 
> Actually, we have more than one - the memory barriers are hidden in outX()
> macros.

You seem to be right.
Here are the results of my investigation:
* IDE DMA on reads just works fine
* IDE DMA on writes does produce corruption of a couple of bytes at offsets
  of multiples of PAGE_SIZE (8k)
* If I limit the DMA SG segments to not cross the page boundaries, DMA write
  seems to work. (I see occasional DMA timeouts though, so it might be
  necessary to limit the SG table length, or maybe to disable all the
  debugging printks currently active.)

Expect a patch soon.

The sg tables are set up correctly, AFAICS.
The only thing which makes me wonder, is why the EOT bit is not set on the
last table entry. But it does not seem to make a difference for the CMD646.

So I  wonder where the bug actually is. Somewhere in hardware; but I wonder
whether the CMD646 or the 2117x (PYXIS/CIA) is to blame. The QLogicISP seems
to happily do BM-DMA, so I'd point to the CMD646. OTOH, the QLogic sits on
the 2nd PCI bus (32bit), which could make a difference as well.

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, DE                                SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: IDE DMA on AXP & barriers

[Ivan Kokshaysky]

On Fri, Dec 07, 2001 at 01:25:05PM +0100, Kurt Garloff wrote:
> So I  wonder where the bug actually is. Somewhere in hardware; but I wonder
> whether the CMD646 or the 2117x (PYXIS/CIA) is to blame. The QLogicISP seems
> to happily do BM-DMA, so I'd point to the CMD646.

Hmm, it seems to be a pyxis bug; the hardware workaround exists, but
I guess that it might be not implemented properly on early miatas.
This also explains why I don't have that problem on lx164 and sx164.
>From pyxis manual:
"A.1 Read Page Problem
 PCI DMA reads that attempt to cross 8K page boundaries cause data corruption
 problems. A fix has been implemented with an Altera 7032 and two Pericom
 PI5C3400 bus switches and a diode."

> OTOH, the QLogic sits on
> the 2nd PCI bus (32bit), which could make a difference as well.

Certainly - the 2nd bus is behind PCI-PCI bridge.

Ivan.

Re: IDE DMA on AXP & barriers

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 2642 bytes --]

Hi Ivan,

On Fri, Dec 07, 2001 at 05:03:41PM +0300, Ivan Kokshaysky wrote:
> On Fri, Dec 07, 2001 at 01:25:05PM +0100, Kurt Garloff wrote:
> > So I  wonder where the bug actually is. Somewhere in hardware; but I wonder
> > whether the CMD646 or the 2117x (PYXIS/CIA) is to blame. The QLogicISP seems
> > to happily do BM-DMA, so I'd point to the CMD646.
> 
> Hmm, it seems to be a pyxis bug; the hardware workaround exists, but
> I guess that it might be not implemented properly on early miatas.
> This also explains why I don't have that problem on lx164 and sx164.
> >From pyxis manual:
> "A.1 Read Page Problem
>  PCI DMA reads that attempt to cross 8K page boundaries cause data corruption
>  problems. A fix has been implemented with an Altera 7032 and two Pericom
>  PI5C3400 bus switches and a diode."

Hey, where did you find that manual? I could not find one at Compaq's web
site.

I wouldn't mind to solder a diode on my board, but I don't know about the
rest ... The software workaround is even preferable, as it helps more people
than just me.

How do I recognize the broken PYXIS in software? (Except for waiting for
your hard disk to be corrupted?)

Unfortunately, I see no 21174 on my PCI bus where I could just check the
revision.
garloff@pws:~ $ /sbin/lspci 
00:03.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 30)
00:04.0 IDE interface: CMD Technology Inc PCI0646 (rev 01)
00:07.0 Non-VGA unclassified device: Intel Corporation 82378IB [SIO ISA Bridge] (rev 43)
00:0c.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2064W [Millennium] (rev 01)
00:14.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 02)
01:09.0 SCSI storage controller: Q Logic ISP1020 (rev 05)

Can I draw conclusions from the revision of the 21152?

Or should I just put an #ifdef CONFIG_ALPHA_PYXIS in my patch?
What about the users of generic alpha kernels?
Or a config option?

Attached is my current version of a workaround. It checks for CMD646, which
should probably be removed, because the bug is in PYXIS according to your
docs. It runs stably since a couple of hours with DMA on, now.


Could people test it? 
Please report back to me whether you have that problem on your PWS ar
another Miata(PYXIS) board.
If the workaround helps you as well, I will prepare a nice patch and submit
it to Linux/Marcelo/Alan.

Regards,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, DE                                SCSI, Security

[-- Attachment #1.2: ide-dma.cmd646.miata.diff --]
[-- Type: text/plain, Size: 1859 bytes --]

--- linux-2.4.16.SuSE-2/drivers/ide/ide-dma.c	Mon Dec  3 01:28:08 2001
+++ linux-2.4.16.SuSE-2.axp/drivers/ide/ide-dma.c	Fri Dec  7 13:24:08 2001
@@ -379,6 +379,21 @@
 			xcount = bcount & 0xffff;
 			if (is_trm290_chipset)
 				xcount = ((xcount >> 2) - 1) << 16;
+#ifdef CONFIG_ALPHA_PYXIS
+			/* Don't cross page boundaries on PYXIS && CMD646 && write */
+			if (HWIF(drive)->chipset == ide_cmd646 && func == ide_dma_write) {
+				if (xcount ==0) 
+					xcount = 0x10000;
+				while ((cur_addr & (PAGE_SIZE-1)) + xcount > PAGE_SIZE) {
+					u32 newcnt = PAGE_SIZE - (cur_addr & (PAGE_SIZE-1));
+					if (count++ >= PRD_ENTRIES)
+						goto use_pio_instead;
+					*table++ = cpu_to_le32(newcnt);
+					*table++ = cpu_to_le32(cur_addr += newcnt);
+					xcount -= newcnt;
+				}
+			}
+#endif
 			if (xcount == 0x0000) {
 				/* 
 				 * Most chipsets correctly interpret a length
@@ -391,12 +406,14 @@
 					goto use_pio_instead;
 
 				*table++ = cpu_to_le32(0x8000);
-				*table++ = cpu_to_le32(cur_addr + 0x8000);
+				*table++ = cpu_to_le32(cur_addr += 0x8000);
 				xcount = 0x8000;
 			}
-			*table++ = cpu_to_le32(xcount);
-			cur_addr += bcount;
+			cur_addr += xcount;
 			cur_len -= bcount;
+			/* KG, 2001-12-07: Mark EOT (bit 31 in length field) */
+			// *table++ = cpu_to_le32(xcount | (cur_len? 0: 0x80000000));
+			*table++ = cpu_to_le32(xcount);
 		}
 
 		sg++;
@@ -654,6 +692,10 @@
 			reading = 1 << 3;
 		case ide_dma_write:
 			SELECT_READ_WRITE(hwif,drive,func);
+#if 0 //def __alpha__
+			if (HWIF(drive)->chipset == ide_cmd646 && func == ide_dma_write)
+				return 1; /* For now, disable DMA writing CMD64x on AXP */
+#endif
 			if (!(count = ide_build_dmatable(drive, func)))
 				return 1;	/* try PIO instead of DMA */
 			outl(hwif->dmatable_dma, dma_base + 4); /* PRD table */

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: IDE DMA on AXP & barriers

[Ivan Kokshaysky]

On Fri, Dec 07, 2001 at 03:48:15PM +0100, Kurt Garloff wrote:
> > Hmm, it seems to be a pyxis bug; the hardware workaround exists, but
> > I guess that it might be not implemented properly on early miatas.
> > This also explains why I don't have that problem on lx164 and sx164.
> > >From pyxis manual:
> > "A.1 Read Page Problem
> >  PCI DMA reads that attempt to cross 8K page boundaries cause data corruption
> >  problems. A fix has been implemented with an Altera 7032 and two Pericom
> >  PI5C3400 bus switches and a diode."
> 
> Hey, where did you find that manual? I could not find one at Compaq's web
> site.

IIRC, few years ago someone posted a link on axp-list, and I picked it up.
Anyway, I've placed it on
ftp://ftp.park.msu.ru/ink/docs/21174_SI.pdf

> How do I recognize the broken PYXIS in software? (Except for waiting for
> your hard disk to be corrupted?)

Put the chip into PCI loopback mode, read some memory (crossing the
page boundary) via direct PCI window and check for corruption -
perhaps this will work.

> Unfortunately, I see no 21174 on my PCI bus where I could just check the
> revision.

Checking the revision won't help - that bug should be fixed with some
off-chip logic.

> Or should I just put an #ifdef CONFIG_ALPHA_PYXIS in my patch?
> What about the users of generic alpha kernels?

#ifdef CONFIG_ALPHA_PYXIS won't work for them.

> Or a config option?

Maybe...
Jay, your opinion? Perhaps you have the info which systems are affected?

Ivan.

Re: IDE DMA on AXP & barriers

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

Hi Ivan, Jay,

[stripped the distributor's lists from Cc: again; I just wanted the patch to
 reach lots of people that can potentially test.]

On Fri, Dec 07, 2001 at 07:43:47PM +0300, Ivan Kokshaysky wrote:
> On Fri, Dec 07, 2001 at 03:48:15PM +0100, Kurt Garloff wrote:
> > Hey, where did you find that manual? I could not find one at Compaq's web
> > site.
> 
> IIRC, few years ago someone posted a link on axp-list, and I picked it up.
> Anyway, I've placed it on
> ftp://ftp.park.msu.ru/ink/docs/21174_SI.pdf

Got it, thanks!

> > How do I recognize the broken PYXIS in software? (Except for waiting for
> > your hard disk to be corrupted?)
> 
> Put the chip into PCI loopback mode, read some memory (crossing the
> page boundary) via direct PCI window and check for corruption -
> perhaps this will work.

I guess the manual will tell me how to do that ...

> > Or should I just put an #ifdef CONFIG_ALPHA_PYXIS in my patch?
> > What about the users of generic alpha kernels?
> 
> #ifdef CONFIG_ALPHA_PYXIS won't work for them.

That's why I'm looking for something better ...
But on a generic kernel, we have to do a number of things, then:
* Detect PYXIS
* Set into PCI loopback
* Do the cross 8k DMA read
* Set flag if corruption

(And even this test is not completely perfect, as only devices on the
 primary PCI bus seem to be affected.)

> > Or a config option?
> 
> Maybe...

Runtime detection is better of course. Think of distributors ...

On the other hand, the workaround does not hurt performance as far as I could
measure. For reading data from disk (i.e. DMAing to memory), the patch does
not do anything. For writing to disk, I make 16 or 17 PDR segments from 4,
but bonnie would not tell me any difference in performance.
So doing it on every PYXIS would also be an option. 

> Jay, your opinion? Perhaps you have the info which systems are affected?

... and how they can be identified.

Thanks,
-- 
Kurt Garloff  <garloff@suse.de>                          Eindhoven, NL
GPG key: See mail header, key servers         Linux kernel development
SuSE GmbH, Nuernberg, DE                                SCSI, Security

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: IDE DMA on AXP & barriers

[Jay Estabrook]

On Fri, Dec 07, 2001 at 06:02:14PM +0100, Kurt Garloff wrote:
>
> > > How do I recognize the broken PYXIS in software? (Except for waiting for
> > > your hard disk to be corrupted?)
> > 
> > Put the chip into PCI loopback mode, read some memory (crossing the
> > page boundary) via direct PCI window and check for corruption -
> > perhaps this will work.
> 
> I guess the manual will tell me how to do that ...

You might be able to adapt/add to Richard's code that probes CIA/PYXIS
to determine if workarounds (for other things :-) are needed.

> That's why I'm looking for something better ...
> But on a generic kernel, we have to do a number of things, then:
> * Detect PYXIS
> * Set into PCI loopback
> * Do the cross 8k DMA read
> * Set flag if corruption
> 
> (And even this test is not completely perfect, as only devices on the
>  primary PCI bus seem to be affected.)

Correct; if you put the problematic card/driver behind the bridge on
MIATA (ie one of the 32-bit slots), the problems go away. No such luck
on LX/SX, but they may have HW workarounds already implemented.

> > Jay, your opinion? Perhaps you have the info which systems are affected?
> 
> ... and how they can be identified.

"Identification" may need to be done via "probe and see"... :-\

IIRC, there was only ever one rev of PYXIS, which has a number of
problems.

The first MIATAs went out with (nearly) no fixes, except for the
console cruft to prevent "unknown" cards from being placed in the
64-bit slots (unkown cards == unknown drivers == possibly susceptible
to the 8K boundary DMA error).

LX and SX introduced off-chip PYXIS fixes, which fixed at least the
8K boundary DMA problem, and some others, but NOT all, most notably
the PCI READ prefetch.

Late MIATAs, aka MIATA-GL, have off-chip logic that fix ALL the PYXIS
problems, AFAIK. These are distinguished from the earlier model by the
presence of a QLogic ISP-1040C(?) SCSI chip built into the motherboard.

--Jay++

-----------------------------------------------------------------------------
Jay A Estabrook                            Alpha Engineering - LINUX Project
Compaq Computer Corp. - MRO1-2/K15         (508) 467-2080
200 Forest Street, Marlboro MA 01752       Jay.Estabrook@compaq.com
-----------------------------------------------------------------------------