From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S966333AbbDWPZd (ORCPT ); Thu, 23 Apr 2015 11:25:33 -0400 Received: from mail-ie0-f171.google.com ([209.85.223.171]:34577 "EHLO mail-ie0-f171.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S965475AbbDWPZa (ORCPT ); Thu, 23 Apr 2015 11:25:30 -0400 Message-ID: <55390EE1.8020304@gmail.com> Date: Thu, 23 Apr 2015 11:25:21 -0400 From: Austin S Hemmelgarn User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Christoph Lameter , Benjamin Herrenschmidt CC: "Paul E. McKenney" , Jerome Glisse , linux-kernel@vger.kernel.org, linux-mm@kvack.org, jglisse@redhat.com, mgorman@suse.de, aarcange@redhat.com, riel@redhat.com, airlied@redhat.com, aneesh.kumar@linux.vnet.ibm.com, Cameron Buschardt , Mark Hairgrove , Geoffrey Gerfin , John McKenna , akpm@linux-foundation.org Subject: Re: Interacting with coherent memory on external devices References: <20150421214445.GA29093@linux.vnet.ibm.com> <20150422000538.GB6046@gmail.com> <20150422131832.GU5561@linux.vnet.ibm.com> <1429756200.4915.19.camel@kernel.crashing.org> In-Reply-To: x-hashcash: 1:21:150423:cl@linux.com::60148c2f82b72daf86a2828801833be5:bf1791bf11bd3ee7 x-hashcash: 1:21:150423:benh@kernel.crashing.org::8109173de48a8139f54a52e13dfc62cf:879825a93b1cb45e x-hashcash: 1:21:150423:paulmck@linux.vnet.ibm.com::df3021d971fb32fa90e47651ba7dea4a:51dc5f66be6c51b3 x-hashcash: 1:21:150423:j.glisse@gmail.com::985c8cd20f44b8c922cf319400495:8d3de96c3a8877a9 x-hashcash: 1:21:150423:linux-kernel@vger.kernel.org::109f27d27d162183d3fa58033b6171f9:d8014ce4319fe12b x-hashcash: 1:21:150423:linux-mm@kvack.org::d4b4dfcfae9deb9a093e6cc6b1a4e74:af5722e0669f1f17 x-hashcash: 1:21:150423:jglisse@redhat.com::c1d2298331e0c8cbb9fdd80357c17d8:849e9f50ea105056 x-hashcash: 1:21:150423:mgorman@suse.de::55af747f63a19cc84061f44fab82b17e:7853188a0cb5d273 x-hashcash: 1:21:150423:aarcange@redhat.com::b2e2541229d0b558d54fbf9228de7bdf:1fcaef7e9502a471 x-hashcash: 1:21:150423:riel@redhat.com::22903c9ce1861b428f0e35925f7b8ad2:6c2f8a593b1f10f6 x-hashcash: 1:21:150423:airlied@redhat.com::8018808754ee6e0eebff84c54d6bea39:59dcdc4f6b58637e x-hashcash: 1:21:150423:aneesh.kumar@linux.vnet.ibm.com::68d99f6f5004406037d774a845bb3081:ea483a8c342eb98c x-hashcash: 1:21:150423:cabuschardt@nvidia.com::e16b2ecc5d9eedb3ec463fb497c5a723:3b28b40894fa983b x-hashcash: 1:21:150423:mhairgrove@nvidia.com::724c50ffee336538493f3430301f160b:abc8e35ff06135c4 x-hashcash: 1:21:150423:ggerfin@nvidia.com::ef6da56ef6dc814198207bc775c6e756:20b0bbe4295b5719 x-hashcash: 1:21:150423:jmckenna@nvidia.com::7823b9058ffaa2b6d429ec7bf7c0cdde:9d2a7385440799e2 x-hashcash: 1:21:150423:akpm@linux-foundation.org::bdbd26214a777b0ba6c7ae1b214930fe:ef904cdbd571035c x-stampprotocols: hashcash:1:17;mbound:0:10:3000:5000 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms010504080506000003020909" X-Antivirus: avast! (VPS 150423-0, 2015-04-23), Outbound message X-Antivirus-Status: Clean Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org This is a cryptographically signed message in MIME format. --------------ms010504080506000003020909 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable On 2015-04-23 10:25, Christoph Lameter wrote: > On Thu, 23 Apr 2015, Benjamin Herrenschmidt wrote: > >> They are via MMIO space. The big differences here are that via CAPI th= e >> memory can be fully cachable and thus have the same characteristics as= >> normal memory from the processor point of view, and the device shares >> the MMU with the host. >> >> Practically what that means is that the device memory *is* just some >> normal system memory with a larger distance. The NUMA model is an >> excellent representation of it. > > I sure wish you would be working on using these features to increase > performance and the speed of communication to devices. > > Device memory is inherently different from main memory (otherwise the > device would be using main memory) and thus not really NUMA. NUMA at le= ast > assumes that the basic characteristics of memory are the same while jus= t > the access speeds vary. GPU memory has very different performance > characteristics and the various assumptions on memory that the kernel > makes for the regular processors may not hold anymore. > You are restricting your definition of NUMA to what the industry=20 constrains it to mean. Based solely on the academic definition of a=20 NUMA system, this _is_ NUMA. In fact, based on the academic definition, = all modern systems could be considered to be NUMA systems, with each=20 level of cache representing a memory only node. Looking at this whole conversation, all I see is two different views on=20 how to present the asymmetric multiprocessing arrangements that have=20 become commonplace in today's systems to userspace. Your model favors=20 performance, while CAPI favors simplicity for userspace. --------------ms010504080506000003020909 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIGuDCC BrQwggScoAMCAQICAxBuVTANBgkqhkiG9w0BAQ0FADB5MRAwDgYDVQQKEwdSb290IENBMR4w HAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmlu ZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZzAeFw0xNTAz MjUxOTM0MzhaFw0xNTA5MjExOTM0MzhaMGMxGDAWBgNVBAMTD0NBY2VydCBXb1QgVXNlcjEj MCEGCSqGSIb3DQEJARYUYWhmZXJyb2luN0BnbWFpbC5jb20xIjAgBgkqhkiG9w0BCQEWE2Fo ZW1tZWxnQG9oaW9ndC5jb20wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCdD/zW 2rRAFCLnDfXpWxU1+ODqRVUgzHvrRO7ADUxRo1CBDc3JSX5TIW2OGmQ3DAKGOACp8Z0sgxMc B05tzAZ/M7m4jajVrwwdVCdrwVGxTdAai7Kwg4ZCVfyMVhcwo8R2eW3QahBx34G0RKumK9sZ ZQSQ+zULAzpY6uz7T1sAk/erMoivRXF6u8WvOsLkOD1F/Xyv1ZccSUG5YeDgZgc0nZUBvyIp zXSHjgWerFkrxEM3y2z/Ff3eL1sgGYecV/I1F+I5S01V7Kclt/qRW10c/4JEGRcI1FmrJBPu BtMYPbg/3Y9LZROYN+mVIFxZxOfrmjfFZ96xt/TaMXo8vcEKtWcNEjhGBjEbfMUEm4aq8ygQ 4MuEcpJc8DJCHBkg2KBk13DkbU2qNepTD6Uip1C+g+KMr0nd6KOJqSH27ZuNY4xqV4hIxFHp ex0zY7mq6fV2o6sKBGQzRdI20FDYmNjsLJwjH6qJ8laxFphZnPRpBThmu0AjuBWE72GnI1oA aO+bs92MQGJernt7hByCnDO82W/ykbVz+Ge3Sax8NY0m2Xdvp6WFDY/PjD9CdaJ9nwQGsUSa N54lrZ2qMTeCI9Vauwf6U69BA42xgk65VvxvTNqji+tZ4aZbarZ7el2/QDHOb/rRwlCFplS/ z4l1f1nOrE6bnDl5RBJyW3zi74P6GwIDAQABo4IBWTCCAVUwDAYDVR0TAQH/BAIwADBWBglg hkgBhvhCAQ0ESRZHVG8gZ2V0IHlvdXIgb3duIGNlcnRpZmljYXRlIGZvciBGUkVFIGhlYWQg b3ZlciB0byBodHRwOi8vd3d3LkNBY2VydC5vcmcwDgYDVR0PAQH/BAQDAgOoMEAGA1UdJQQ5 MDcGCCsGAQUFBwMEBggrBgEFBQcDAgYKKwYBBAGCNwoDBAYKKwYBBAGCNwoDAwYJYIZIAYb4 QgQBMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AuY2FjZXJ0Lm9y ZzAxBgNVHR8EKjAoMCagJKAihiBodHRwOi8vY3JsLmNhY2VydC5vcmcvcmV2b2tlLmNybDA0 BgNVHREELTArgRRhaGZlcnJvaW43QGdtYWlsLmNvbYETYWhlbW1lbGdAb2hpb2d0LmNvbTAN BgkqhkiG9w0BAQ0FAAOCAgEAGvl7xb42JMRH5D/vCIDYvFY3dR2FPd5kmOqpKU/fvQ8ovmJa p5N/FDrsCL+YdslxPY+AAn78PYmL5pFHTdRadT++07DPIMtQyy2qd+XRmz6zP8Il7vGcEDmO WmMLYMq4xV9s/N7t7JJp6ftdIYUcoTVChUgilDaRWMLidtslCdRsBVfUjPb1bF5Ua31diKDP e0M9/e2CU36rbcTtiNCXhptMigzuL3zJXUf2B9jyUV8pnqNEQH36fqJ7YTBLcpq3aYa2XbAH Hgx9GehJBIqwspDmhPCFZ/QmqUXCkt+XfvinQ2NzKR6P3+OdYbwqzVX8BdMeojh7Ig8x/nIx mQ+/ufstL1ZYp0bg13fyK/hPYSIBpayaC76vzWovkIm70DIDRIFLi20p/qTd7rfDYy831Hjm +lDdCECF9bIXEWFk33kA97dgQIMbf5chEmlFg8S0e4iw7LMjvRqMX3eCD8GJ2+oqyZUwzZxy S0Mx+rBld5rrN7LsXwZ671HsGqNeYbYeU25e7t7/Gcc6Bd/kPfA+adEuUGFcvUKH3trDYqNq 6mOkAd8WO/mQadlc3ztS++XDMhmIpfBre9MPAr6usqf+wc+R8Nk9KLK39kEgrqVfzc/fgf8L MaD4rHnusdg4gca6Yi+kNrm99anw7SwaBrBvULYBp7ixNRUhaYiNW4YjTrYxggShMIIEnQIB ATCBgDB5MRAwDgYDVQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5v cmcxIjAgBgNVBAMTGUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEW EnN1cHBvcnRAY2FjZXJ0Lm9yZwIDEG5VMAkGBSsOAwIaBQCgggH1MBgGCSqGSIb3DQEJAzEL BgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTE1MDQyMzE1MjUyMVowIwYJKoZIhvcNAQkE MRYEFHv7W7/H3OoJZEzy3M3wVmPBvhMuMGwGCSqGSIb3DQEJDzFfMF0wCwYJYIZIAWUDBAEq MAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzAOBggqhkiG9w0DAgICAIAwDQYIKoZIhvcNAwIC AUAwBwYFKw4DAgcwDQYIKoZIhvcNAwICASgwgZEGCSsGAQQBgjcQBDGBgzCBgDB5MRAwDgYD VQQKEwdSb290IENBMR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMT GUNBIENlcnQgU2lnbmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2Fj ZXJ0Lm9yZwIDEG5VMIGTBgsqhkiG9w0BCRACCzGBg6CBgDB5MRAwDgYDVQQKEwdSb290IENB MR4wHAYDVQQLExVodHRwOi8vd3d3LmNhY2VydC5vcmcxIjAgBgNVBAMTGUNBIENlcnQgU2ln bmluZyBBdXRob3JpdHkxITAfBgkqhkiG9w0BCQEWEnN1cHBvcnRAY2FjZXJ0Lm9yZwIDEG5V MA0GCSqGSIb3DQEBAQUABIICAJgmceESJ6F394pUTUc+0x+vMuoZaO+3qYAD90OmwiHUpigz 5oBmA5Fwf+/5Mrv5vm1au80VUp+xj4x9EMsa9d24I9YHtH88iARcswVTD0QQGozmN6e+uSu0 TDAmqTtX5opWPirY4h54RtCY9uqtVBTfADoN/GWyDfdVND61OPWG2N6MvGOP1ecSR6xMy89/ 5Ug+vZMK9HU1Pxc+H+nzkiG3aD9tU9P/PZsw5sborGgaPkf3vQlQ0MNRDlzholiu4B6oIOEn z8NVqWrhC4ldZbd9WR+TYyTSpPyJKMC1gQ4N11qoV+4CuJ6Tc6i+zP3aCq1HSUBatWg4jPq8 AF++hp7bcEGh5TYX1AjBy03aSAywK98haRFoBRzEpdWiXS9exH4MDe2ZZj1bWwNYfCqbwA1G nnvzJZbEcQf9XYPNlDEgVcGHhFCC5u2JGEOYGLiUUoSAJsewbzy5EMgi7GgG8PWCP9pMM6hj SxdpE1TuPpPq8a6Wm89bhcqDq8ACOY/o62MNKEy1Ku+Z2SzKjxSPbW58qmvhWhYIjvIwmBlG EWy8RxBoOhlq2J12t9S6ktUFsEbpeZGqJ6Yv9Vm0S5UEX1S1nL766X6t5cj9bVbm0v6mKj3n SBdY+/y3TzzPzBnJAYKE3euChpD6/kdL+idmmvek6sx4cpz9dVcTRabwsPGWAAAAAAAA --------------ms010504080506000003020909--