mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Mark Haverkamp <markh@osdl.org>
To: James.Smart@Emulex.Com
Cc: Doug Maxey <dwm@maxeymade.com>, Olof Johansson <olof@lixom.net>,
	"linuxppc64-dev@ozlabs.org" <linuxppc64-dev@ozlabs.org>,
	linux-kernel <linux-kernel@vger.kernel.org>,
	linux-scsi <linux-scsi@vger.kernel.org>
Subject: Re: iommu_alloc failure and panic
Date: Tue, 31 Jan 2006 13:33:00 -0800	[thread overview]
Message-ID: <1138743180.15732.15.camel@markh3.pdx.osdl.net> (raw)
In-Reply-To: <43DF691E.1020008@emulex.com>

On Tue, 2006-01-31 at 08:41 -0500, James Smart wrote:
> >> 2) The emulex driver has been prone to problems in the past where it's
> >> been very aggressive at starting DMA operations, and I think it can
> >> be avoided with tuning. What I don't know is if it's because of this,
> >> or simply because of the large number of targets you have. Cc:ing James
> >> Smart.
> 
> I don't have data points for the 2.6 kernel, but I can comment on what I
> have seen on the 2.4 kernel.
> 
> The issue that I saw on the 2.4 kernel was that the pci dma alloc routine
> was inappropriately allocating from the dma s/g maps. On systems with less
> than 4Gig of memory, or on those with no iommmu (emt64), the checks around
> adapter-supported dma masks were off (I'm going to be loose in terms to not
> describe it in detail). The result was, although the adapter could support
> a fully 64bit address and/or although the physical dma address would be under
> 32-bits, the logic forced allocation from the mapped dma pool. On some
> systems, this pool was originally only 16MB. Around 2.4.30, the swiotlb was
> introduced, which reduced issue, but unfortunately, still never solved the
> allocation logic. It fails less as the swiotlb simply had more space.
> As far as I know, this problem doesn't exist in the 2.6 kernel. I'd have to
> go look at the dma map functions to make sure.
> 
> Why was the lpfc driver prone to the dma map exhaustion failures ? Due to the
> default # of commands per lun and max sg segments reported by the driver to
> the scsi midlayer, the scsi mid-layer's preallocation of dma maps for commands
> for each lun, and the fact that our FC configs were usually large, had lots
> of luns, and replicated the resources for each path to the same storage.
> 
> Ultimately, what I think is the real issue here is the way the scsi mid-layer
> is preallocating dma maps for the luns. 16000 luns is a huge number.
> Multiply this by a max sg segment count of 64 by the driver, and a number
> between 3 and 30 commands per lun, and you can see the numbers. Scsi does do
> some interesting allocation algorithms once it hits an allocation failure.
> One side effect of this is that it is fairly efficient at allocating the
> bulk of the dma pool.

James,

Thanks for the information.  I tried loading the lpfc driver with
lpfc_lun_queue_depth=1 and haven't seen iommu_alloc failures.  I'm still
curious why the alloc failures lead to a panic though.

Mark.


> 
> -- james s
-- 
Mark Haverkamp <markh@osdl.org>


  reply	other threads:[~2006-01-31 21:33 UTC|newest]

Thread overview: 13+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2006-01-27 16:57 Mark Haverkamp
2006-01-27 20:40 ` Olof Johansson
2006-01-27 22:39   ` Mark Haverkamp
2006-01-27 23:34     ` Olof Johansson
2006-01-27 23:37       ` Mark Haverkamp
2006-01-30 15:32       ` Mark Haverkamp
2006-01-30 23:13         ` Olof Johansson
2006-01-31  1:18           ` Doug Maxey
2006-01-31 13:41             ` James Smart
2006-01-31 21:33               ` Mark Haverkamp [this message]
2006-01-27 23:48 ` Anton Blanchard
2006-01-28  0:00   ` Mark Haverkamp
2006-01-28  1:19     ` Anton Blanchard

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1138743180.15732.15.camel@markh3.pdx.osdl.net \
    --to=markh@osdl.org \
    --cc=James.Smart@Emulex.Com \
    --cc=dwm@maxeymade.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=linuxppc64-dev@ozlabs.org \
    --cc=olof@lixom.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®