From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8BEF91586CB for ; Tue, 23 Jul 2024 18:24:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721759097; cv=none; b=jU63dofHVApOyQGWcfcsLuaBN2AAD+Ko+F/zAiirbiCm4w3wUEuK0i2LY/IKQdTi5y5ZGZ8bj9aDiu+Cc9i1K2iRdvo9FQJbie9cEp8R9ueqzAr2sUN3AqHsFskeswt5WkCmDJTtMv/UTmREnNbfazmGEGEA7qvRnvNClfDf7MY= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721759097; c=relaxed/simple; bh=z4klGlA66rZwp0B0TEsP74siGqi9zdcPyuDjYgQCAcY=; h=Message-ID:Date:MIME-Version:Subject:From:To:Cc:References: In-Reply-To:Content-Type; b=SL3/P84+0bKdRa3INIeFFD+2AmNPdsHCkwfR4dX4BYe+cmkpMHUfUUZexccz4N1YXCsaaodvZ9b7KvjbQimiSHhmawxtfzysMniPbLRw6qF6ufjTGnQn6f5hxoh3n1hOoFS2XNrXZ62e/POhAsIyBU+y/lUw3/vTXvj8A1HbQ+U= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=OnQ0Dnby; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="OnQ0Dnby" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721759094; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=mlSldsjWGkt137rHxqPtPFE0Ch/bG3EUj3dygNmEleA=; b=OnQ0Dnby7ZkD7DUBzGe4ZLwiXPAB1UNfZ95jmD5ZpIBeX/dGBF9tvVMsg28Eack9z1DEzr BZ55M7RGuiVXMC00e6j/PsYW9mmaJQcih2DxuJKdfxhn3yxIMBrXrl706c5/hSfcqdAvMq xwwlnm+2N9e3XmvZSVMI1m0TLx7b3Ec= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-615-jZMsveuwNTOgOb3_k3pd0Q-1; Tue, 23 Jul 2024 14:24:53 -0400 X-MC-Unique: jZMsveuwNTOgOb3_k3pd0Q-1 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4265d3bf59dso41610905e9.3 for ; Tue, 23 Jul 2024 11:24:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721759092; x=1722363892; h=content-transfer-encoding:in-reply-to:organization:autocrypt :content-language:references:cc:to:from:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=mlSldsjWGkt137rHxqPtPFE0Ch/bG3EUj3dygNmEleA=; b=NvCdwfm1+Q8t7cKFt+c2VaJHx9sAdBvXLj2frlOjA73byoM85TdDux/93es066ZeEb d0/e03DRYCaIJg8c0gqoR31mCi9+9xUqjGz7/zFLP35W4hHeZXC9MamZf7bTavWhtkGb xDVKcmaK92YluPGeXx4haJQ0Pi4fb+LpsF4ZsCCqvw2vrBecW+EZOgdzPIVUFDE245oh BHdfg/oTe7QZgTdgm2Ycb7OJxhvGn9aJ8RECOmok4BrstRAdcuadjL8vSw6CsGBXMTtZ YnV/5FmGp8RkEzkJ12JD9XjMrXZ8RGWxgnNj41bxECS9eXD7tGQTOyEuJ5HlcJPd7M6N TbEw== X-Forwarded-Encrypted: i=1; AJvYcCXozxcktZIt3V+E+xAnX/EaI4V/fuIDy5xU+QYZKSGKJYLqCRG++3yI8WUV6E6BS7Z/wSThjh0CZXeXm89N3io1O6c2kwaBtmR/3zG5 X-Gm-Message-State: AOJu0YxPWDz3azV0Y7ZHgheNvPPVkhhtJb63v36jfNv2BH1chg2knAEe InyJ6XHzvkz4qle+PJK0lFxGKyjMZ/ZhSpE+2N2bdyjJOQq+q03zc2FhhUkbM9np0TK1ZvmgYj3 CFWu1LcuO4eOqGC9zz03S1BMKSlkPCzNJcl+YjIc5VPhQp3dVYPPwnBbQS+lnMQ== X-Received: by 2002:a05:600c:1c13:b0:426:5ef5:bcb1 with SMTP id 5b1f17b1804b1-427f7a15278mr2493765e9.6.1721759091949; Tue, 23 Jul 2024 11:24:51 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHq7GXtYAKFTz+qGMMS1RUEa8XcH1p15cy6QfGw/2Plhyb1cls798ESEkmZL12VE1vSOh74Nw== X-Received: by 2002:a05:600c:1c13:b0:426:5ef5:bcb1 with SMTP id 5b1f17b1804b1-427f7a15278mr2493625e9.6.1721759091497; Tue, 23 Jul 2024 11:24:51 -0700 (PDT) Received: from ?IPV6:2003:cb:c72f:ed00:9dc2:1adb:d133:4434? (p200300cbc72fed009dc21adbd1334434.dip0.t-ipconnect.de. [2003:cb:c72f:ed00:9dc2:1adb:d133:4434]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-427d6901781sm182893445e9.14.2024.07.23.11.24.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 23 Jul 2024 11:24:51 -0700 (PDT) Message-ID: Date: Tue, 23 Jul 2024 20:24:49 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] mm: fix maxnode for mbind(), set_mempolicy() and migrate_pages() From: David Hildenbrand To: Jerome Glisse Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, stable@vger.kernel.org, Andi Kleen References: <20240720173543.897972-1-jglisse@google.com> <0c390494-e6ba-4cde-aace-cd726f2409a1@redhat.com> <6be6453a-15ce-4305-9a7c-a66e57564785@redhat.com> Content-Language: en-US Autocrypt: addr=david@redhat.com; keydata= xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9 M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3 AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8 kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt WNyWQQ== Organization: Red Hat In-Reply-To: <6be6453a-15ce-4305-9a7c-a66e57564785@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 23.07.24 19:37, David Hildenbrand wrote: > On 23.07.24 18:33, Jerome Glisse wrote: >> On Mon, 22 Jul 2024 at 06:09, David Hildenbrand wrote: >>> >>> On 20.07.24 19:35, Jerome Glisse wrote: >>>> Because maxnode bug there is no way to bind or migrate_pages to the >>>> last node in multi-node NUMA system unless you lie about maxnodes >>>> when making the mbind, set_mempolicy or migrate_pages syscall. >>>> >>>> Manpage for those syscall describe maxnodes as the number of bits in >>>> the node bitmap ("bit mask of nodes containing up to maxnode bits"). >>>> Thus if maxnode is n then we expect to have a n bit(s) bitmap which >>>> means that the mask of valid bits is ((1 << n) - 1). The get_nodes() >>>> decrement lead to the mask being ((1 << (n - 1)) - 1). >>>> >>>> The three syscalls use a common helper get_nodes() and first things >>>> this helper do is decrement maxnode by 1 which leads to using n-1 bits >>>> in the provided mask of nodes (see get_bitmap() an helper function to >>>> get_nodes()). >>>> >>>> The lead to two bugs, either the last node in the bitmap provided will >>>> not be use in either of the three syscalls, or the syscalls will error >>>> out and return EINVAL if the only bit set in the bitmap was the last >>>> bit in the mask of nodes (which is ignored because of the bug and an >>>> empty mask of nodes is an invalid argument). >>>> >>>> I am surprised this bug was never caught ... it has been in the kernel >>>> since forever. >>> >>> Let's look at QEMU: backends/hostmem.c >>> >>> /* >>> * We can have up to MAX_NODES nodes, but we need to pass maxnode+1 >>> * as argument to mbind() due to an old Linux bug (feature?) which >>> * cuts off the last specified node. This means backend->host_nodes >>> * must have MAX_NODES+1 bits available. >>> */ >>> >>> Which means that it's been known for a long time, and the workaround >>> seems to be pretty easy. >>> >>> So I wonder if we rather want to update the documentation to match reality. >> >> [Sorry resending as text ... gmail insanity] >> >> I think it is kind of weird if we ask to supply maxnodes+1 to work >> around the bug. If we apply this patch qemu would continue to work as >> is while fixing users that were not aware of that bug. So I would say >> applying this patch does more good. Long term qemu can drop its >> workaround or keep it for backward compatibility with old kernel. > > Not really, unfortunately. The thing is that it requires a lot more > effort to sense support than simply pass maxnodes+1. So unless you know > exactly on which minimum kernel version your software runs (barely > happens), you will simply apply the workaround. > > I would assume that each and every sane user out there does that > already, judging that even that QEMU code is 10 years old (!). > > In any case, we have to document that behavior that existed since the > very beginning. Because it would be even *worse* if someone would > develop against a new kernel and would get a bunch of bug reports when > running on literally every old kernel out there :) > > So my best guess is that long-term it will create more issues when we > change the behavior ... but in any case we have to update the man pages. I'll add a pointer to a discussion from 20 (!) year ago [1]. The conclusion [2] / request from Andi there was that N65 is the right thing to do, and we (re)added the --maxnode. This was when this code was introduced. Assuming MAX_NODES is 128, we have to pass in 129 -- one more than the number of bits. So the man page might be actively misleading (unless I misinterpret it): "nodemask points to a bit mask of nodes containing up to maxnode bits. The bit mask size is rounded to the next multiple of sizeof(unsigned long), but the kernel will use bits only up to maxnode." That documentation should be fixed. [1] https://lkml.indiana.edu/hypermail/linux/kernel/0409.1/1538.html [2] https://lkml.indiana.edu/hypermail/linux/kernel/0409.1/1569.html -- Cheers, David / dhildenb