From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753952AbbLRW1l (ORCPT ); Fri, 18 Dec 2015 17:27:41 -0500 Received: from mail-pa0-f42.google.com ([209.85.220.42]:34953 "EHLO mail-pa0-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751441AbbLRW13 (ORCPT ); Fri, 18 Dec 2015 17:27:29 -0500 From: Douglas Anderson To: Russell King Cc: Robin Murphy , Tomasz Figa , Marek Szyprowski , Pawel Osciak , Dmitry Torokhov , Douglas Anderson , will.deacon@arm.com, akpm@linux-foundation.org, carlo@caione.org, laurent.pinchart+renesas@ideasonboard.com, mike.looijmans@topic.nl, penguin-kernel@i-love.sakura.ne.jp, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/2] ARM: dma-mapping: sort the pages after allocation Date: Fri, 18 Dec 2015 14:27:02 -0800 Message-Id: <1450477622-30948-2-git-send-email-dianders@chromium.org> X-Mailer: git-send-email 2.6.0.rc2.230.g3dd15c0 In-Reply-To: <1450477622-30948-1-git-send-email-dianders@chromium.org> References: <1450477622-30948-1-git-send-email-dianders@chromium.org> Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org After doing allocation, make one last-ditch effort to get contiguous regions of pages to optimize TLB usage. This is a rather simplistic approach that could be later optimized, but it doesn't hurt and should only have the opportunity to help. >>From my testing the sort took less than 400us for a 4MB allocation. That's much faster than the actual allocation which was more than a millisecond even in the fastest case (and was often several hundred ms). Signed-off-by: Douglas Anderson --- Changes in v2: - Sort patch new for v2 (and optional if people hate it). arch/arm/mm/dma-mapping.c | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c index 9887d432cf1f..d1b3d3e6fe47 100644 --- a/arch/arm/mm/dma-mapping.c +++ b/arch/arm/mm/dma-mapping.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -1122,6 +1123,21 @@ static inline void __free_iova(struct dma_iommu_mapping *mapping, spin_unlock_irqrestore(&mapping->lock, flags); } +static int cmp_pfns(const void *a, const void *b) +{ + unsigned long a_pfn; + unsigned long b_pfn; + + a_pfn = page_to_pfn(*(struct page **)a); + b_pfn = page_to_pfn(*(struct page **)b); + + if (a_pfn < b_pfn) + return -1; + else if (a_pfn > b_pfn) + return 1; + return 0; +} + /* We'll try 2M, 1M, 64K, and finally 4K; array must end with 0! */ static const int iommu_order_array[] = { 9, 8, 4, 0 }; @@ -1133,6 +1149,7 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, int array_size = count * sizeof(struct page *); int i = 0; int order_idx = 0; + int first_order_zero = -1; if (array_size <= PAGE_SIZE) pages = kzalloc(array_size, GFP_KERNEL); @@ -1171,6 +1188,7 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, /* Drop down when we get small */ if (__fls(count) < order) { order_idx++; + /* Don't update first_order_zero; no need to sort end */ continue; } @@ -1181,6 +1199,8 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, /* Go down a notch at first sign of pressure */ if (!pages[i]) { order_idx++; + if (iommu_order_array[order_idx] == 0) + first_order_zero = i; continue; } } else { @@ -1201,6 +1221,26 @@ static struct page **__iommu_alloc_buffer(struct device *dev, size_t size, count -= 1 << order; } + /* + * If we folded under memory pressure, try one last ditch event to get + * contiguous pages via sorting. Under testing this sometimes helped + * get a few more contiguous pages and didn't cost much compared to + * the above allocations. + * + * Note that we only sort the order zero pages so that we don't mess + * up the higher order allocations by sticking small pages in between + * them. + * + * If someone wanted to optimize this more, they could insert extra + * (out of order) single pages in places to help keep virtual and + * physical pages aligned with each other. As it is we often get + * lucky and get the needed alignment but we're not guaranteed. + */ + if (first_order_zero >= 0) + sort(pages + first_order_zero, + (size >> PAGE_SHIFT) - first_order_zero, sizeof(*pages), + cmp_pfns, NULL); + return pages; error: while (i--) -- 2.6.0.rc2.230.g3dd15c0