From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Google-Smtp-Source: AH8x224x6BGI7U4N/K9ZQ7sgaaEZekJjCBHgZyUBdQp6VhiPDzTGa46X36ooyu3czf9Q7cv2an/N ARC-Seal: i=1; a=rsa-sha256; t=1516751664; cv=none; d=google.com; s=arc-20160816; b=hkaIQ7DHe8AmL2UuSdNLwHRwH/AZ0Sqv4YU1tnax0olmBvqu2yBb/7MwAUpIoNR7NN 1ulxbGAgXP01eZEqAoxzmpPnT5fL9Cgva7CyJwalxiRURMegsPPdhARz+0wjfuDEYEl0 UyPq5FvSjo8OOV4oSww0peq9vC7Gjxbmn31s/6Enav9xWjxgLkYZXroPznAn8Eum1vYh iJ7RbzWSo1rbCDpRRFXMVEuyG84y0YCACQD/2ObOmUD4LPlQmYCZKaYb3fhO2owM07Ps 3eanyMGh2zaEFUkweS9A3Udm3zvXRZ7a8yVK4seCOJRKTHiQ+UsXMrFqdJXh/m96z8pc /MEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:dkim-signature:arc-authentication-results; bh=zALOIuxVj77tGxE/WR0FpuP01ElcUGSdjjbLLeKmdhc=; b=lhQGRkWI5F2NwV9YLxRA0DKxN/JFWJl+Xmr4Vc2Aa/9OTmhlKPVevDbLJYG5ngWrms o0/17uI8UKXxcgvA6ORmIpzAuVqjeeJIbC8wR/d4QoczRxAi/kkZljC7p7GDVB1rot/s UsqqRNTQ71mVOoGHRybwHvzO+vzLom4w76gXJQ/UrrLarJTE51e7JiNPUIs4NByJcYQS kOtkjyhVa5Ba7d7fginAWlySJ5tc9wwpwQECwUkWB1gihCUAtwaaubW7LJFfCwhcZ5Uk zPtW1k/GYzYHqZzmfog/aMW3UmH5rflJuUYSmsJ9cQ2Bct05TySNUDTaiRS4xoWzYIp8 wk0Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2017-10-26 header.b=ITERGwYe; spf=pass (google.com: domain of henry.willard@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=henry.willard@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2017-10-26 header.b=ITERGwYe; spf=pass (google.com: domain of henry.willard@oracle.com designates 141.146.126.79 as permitted sender) smtp.mailfrom=henry.willard@oracle.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com From: Henry Willard To: akpm@linux-foundation.org Cc: mgorman@suse.de, kstewart@linuxfoundation.org, zi.yan@cs.rutgers.edu, pombredanne@nexb.com, aarcange@redhat.com, gregkh@linuxfoundation.org, aneesh.kumar@linux.vnet.ibm.com, kirill.shutemov@linux.intel.com, jglisse@redhat.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2] mm: numa: Do not trap faults on shared data section pages. Date: Tue, 23 Jan 2018 15:53:37 -0800 Message-Id: <1516751617-7369-1-git-send-email-henry.willard@oracle.com> X-Mailer: git-send-email 1.8.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Proofpoint-Virus-Version: vendor=nai engine=5900 definitions=8783 signatures=668655 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=1 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 mlxscore=0 mlxlogscore=580 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1801230305 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: =?utf-8?q?1590429393267653540?= X-GMAIL-MSGID: =?utf-8?q?1590429393267653540?= X-Mailing-List: linux-kernel@vger.kernel.org List-ID: Workloads consisting of a large number of processes running the same program with a very large shared data segment may experience performance problems when numa balancing attempts to migrate the shared cow pages. This manifests itself with many processes or tasks in TASK_UNINTERRUPTIBLE state waiting for the shared pages to be migrated. The program listed below simulates the conditions with these results when run with 288 processes on a 144 core/8 socket machine. Average throughput Average throughput Average throughput with numa_balancing=0 with numa_balancing=1 with numa_balancing=1 without the patch with the patch --------------------- --------------------- --------------------- 2118782 2021534 2107979 Complex production environments show less variability and fewer poorly performing outliers accompanied with a smaller number of processes waiting on NUMA page migration with this patch applied. In some cases, %iowait drops from 16%-26% to 0. // SPDX-License-Identifier: GPL-2.0 /* * Copyright (c) 2017 Oracle and/or its affiliates. All rights reserved. */ #include #include #include #include int a[1000000] = {13}; int main(int argc, const char **argv) { int n = 0; int i; pid_t pid; int stat; int *count_array; int cpu_count = 288; long total = 0; struct timeval t1, t2 = {(argc > 1 ? atoi(argv[1]) : 10), 0}; if (argc > 2) cpu_count = atoi(argv[2]); count_array = mmap(NULL, cpu_count * sizeof(int), (PROT_READ|PROT_WRITE), (MAP_SHARED|MAP_ANONYMOUS), 0, 0); if (count_array == MAP_FAILED) { perror("mmap:"); return 0; } for (i = 0; i < cpu_count; ++i) { pid = fork(); if (pid <= 0) break; if ((i & 0xf) == 0) usleep(2); } if (pid != 0) { if (i == 0) { perror("fork:"); return 0; } for (;;) { pid = wait(&stat); if (pid < 0) break; } for (i = 0; i < cpu_count; ++i) total += count_array[i]; printf("Total %ld\n", total); munmap(count_array, cpu_count * sizeof(int)); return 0; } gettimeofday(&t1, 0); timeradd(&t1, &t2, &t1); while (timercmp(&t2, &t1, <)) { int b = 0; int j; for (j = 0; j < 1000000; j++) b += a[j]; gettimeofday(&t2, 0); n++; } count_array[i] = n; return 0; } This patch changes change_pte_range() to skip shared copy-on-write pages when called from change_prot_numa(). NOTE: change_prot_numa() is nominally called from task_numa_work() and queue_pages_test_walk(). task_numa_work() is the auto NUMA balancing path, and queue_pages_test_walk() is part of explicit NUMA policy management. However, queue_pages_test_walk() only calls change_prot_numa() when MPOL_MF_LAZY is specified and currently that is not allowed, so change_prot_numa() is only called from auto NUMA balancing. In the case of explicit NUMA policy management, shared pages are not migrated unless MPOL_MF_MOVE_ALL is specified, and MPOL_MF_MOVE_ALL depends on CAP_SYS_NICE. Currently, there is no way to pass information about MPOL_MF_MOVE_ALL to change_pte_range. This will have to be fixed if MPOL_MF_LAZY is enabled and MPOL_MF_MOVE_ALL is to be honored in lazy migration mode. task_numa_work() skips the read-only VMAs of programs and shared libraries. V2: - Combined patch and cover letter - Added note about applicability of MPOL_MF_MOVE_ALL Signed-off-by: Henry Willard Reviewed-by: Håkon Bugge Reviewed-by: Steve Sistare steven.sistare@oracle.com Acked-by: Mel Gorman --- mm/mprotect.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/mprotect.c b/mm/mprotect.c index ec39f730a0bf..fbbb3ab70818 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -84,6 +84,11 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd, if (!page || PageKsm(page)) continue; + /* Also skip shared copy-on-write pages */ + if (is_cow_mapping(vma->vm_flags) && + page_mapcount(page) != 1) + continue; + /* Avoid TLB flush if possible */ if (pte_protnone(oldpte)) continue; -- 1.8.3.1