From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0B3141448ED for ; Fri, 23 Aug 2024 23:43:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724456606; cv=none; b=Apx0rImWYZpGIX0iIpOfLhwoiN0Pw3ZXZ1D4R8AlZzCC2kPaYjac1HNoQDy5m7gaevilDN3+58glXl4DBRiMVIfT3qs3KXGcVYRD/G0f6OZNKV4kJ4jeTa3rKgTrp0DrIO8Mkfl5NYAhmWzo/Qslz+pyrw28M/pACF3D0GemcKw= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724456606; c=relaxed/simple; bh=YOg/jB7ng9Pg4C5FxhkAA+bHBNHnoTpo/cxYHVtJNIE=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=USqKsdaP0wRoOsCEbA7ZrSaXqnwgdhMnaFMmbCjKUQ1hP0FyV/ztlTibZl9yEOdFv9DI525YtMauriaq2aAQZOvUGydzJNDE04XIEfhhLamH5aMKx9jW0Aj2W/X7X9eklSvOqfC3btrAvshYBoUOO8UZgtQwoaxvJCuHRsVA8To= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MH141oXz; arc=none smtp.client-ip=209.85.208.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MH141oXz" Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5c07eebf29eso1039819a12.2 for ; Fri, 23 Aug 2024 16:43:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1724456603; x=1725061403; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=R7I1uBnhEhOKBlQUhS7kHGTlUcfRgLn9WhZ/VYd/7F0=; b=MH141oXzcDaD2cOZZDkGCADiY/9ZVcR4qCIIapI+CmkN0pBB+KvsjuiId+eHgvwdNA 4fhU5RvfLUccg0DISp8UPPAZh3PnVrjFVOx4IWaym0UyYSNpkVndXdWhKmAaUHmHd/mI /FvUlJpmE+jUZVu8rmnAFfb5UZplLmVUy2OPVv03JipqBHQfb+I8EOIpdnULFd5cqk9U fTdlZNf1pmwPm4OxUGR2JHsN5QLKJ+gUXh0rHgWhFmYGxmNUc/4RZ15Z60UMDxf1+Ygu OH/BciExo/AEK/dizTMX63upbmDJPzBuImNVq4+MWVZPTnvTXHjfoE4dv+f+r9a0QoKi S1Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724456603; x=1725061403; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=R7I1uBnhEhOKBlQUhS7kHGTlUcfRgLn9WhZ/VYd/7F0=; b=Ty5CnvsPDbp0dPCpp0yuAgT02/1VH1wuSI2LRjocfJw5MKtXh7MgCcMrD90ZipCpXr pSl3BKVLwMBRfDTTQgK7ZJN+2ykslqI87MbEIordE0FgS4AGrr2w/h0UYHXmKnnOMYVI thHSYB5thf5fyJPahOVbnh26C/uHUincj7ZnfJs/oEZQ0+yS30ATEjZorcBPtM6AqbMp eS4OVaFqRRyJXsEagONZrY2bBwLDY+2ZuK3HoAl1bs8K12oj0cKOAfoP+qyhWcV+eEd+ D+JF7vlMEI6qpJ9icDOUq5H3ez2EbP3eHbiVJiqJSwV3FNsxqXdmMGX3meQAt7ececQK jB0Q== X-Forwarded-Encrypted: i=1; AJvYcCVumclue2OyBEXWfk8Mds9SjK30u6Imot9x58Uq/rogHvUBAqUELpLYT38oXVp1IhsSviaNwg9iBKnqTaQ=@vger.kernel.org X-Gm-Message-State: AOJu0YwBlB+ZJ4Oi6ZkHS9GJF6YaTAPwFf+ojJJx85I7g3QqtPzm3xic g+dW5Evh+5ACpjJ0JAzEs6HoBs7g6WYzmEmC6758MTBdn0ZOwhC8 X-Google-Smtp-Source: AGHT+IHzF6G4QSBC0eUfzD5aTYYLgTzEdHxc9asDp8KMSoScnuOpfvySF2Fd0lo+2DSjcM42JyCD5g== X-Received: by 2002:a05:6402:1ec8:b0:5bf:1f8:9493 with SMTP id 4fb4d7f45d1cf-5c0891a8338mr3118257a12.29.1724456602855; Fri, 23 Aug 2024 16:43:22 -0700 (PDT) Received: from shift.daheim (p200300d5ff191e0050f496fffe46beef.dip0.t-ipconnect.de. [2003:d5:ff19:1e00:50f4:96ff:fe46:beef]) by smtp.gmail.com with ESMTPSA id 4fb4d7f45d1cf-5c04a3e89ebsm2617845a12.43.2024.08.23.16.43.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Aug 2024 16:43:22 -0700 (PDT) Received: from localhost.daheim ([127.0.0.1]) by shift.daheim with esmtp (Exim 4.98) (envelope-from ) id 1shdw2-00000002QcG-206q; Sat, 24 Aug 2024 01:43:21 +0200 Message-ID: <90b971f6-16d6-4a9a-9dc5-40b291376952@gmail.com> Date: Sat, 24 Aug 2024 01:43:21 +0200 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] powerpc: warn on emulation of dcbz instruction in kernel mode To: Segher Boessenkool , Christoph Hellwig Cc: LEROY Christophe , Benjamin Herrenschmidt , Paul Mackerras , Michael Ellerman , "linux-kernel@vger.kernel.org" , "linuxppc-dev@lists.ozlabs.org" , Stan Johnson , Finn Thain References: <2e3acfe63d289c6fba366e16973c9ab8369e8b75.1631803922.git.christophe.leroy@csgroup.eu> <17fa6450-6613-4c34-804b-e47246e7b39c@isd.uni-stuttgart.de> <9dbf73fe-a459-4956-8dbc-e919d9728f5e@cs-soprasteria.com> <20240822053238.GA2028@lst.de> <20240823130600.GI28254@gate.crashing.org> <20240823135459.GA28487@lst.de> <20240823191924.GK28254@gate.crashing.org> Content-Language: en-US From: Christian Lamparter In-Reply-To: <20240823191924.GK28254@gate.crashing.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 8/23/24 9:19 PM, Segher Boessenkool wrote: > Hi! > > On Fri, Aug 23, 2024 at 03:54:59PM +0200, Christoph Hellwig wrote: >> On Fri, Aug 23, 2024 at 08:06:00AM -0500, Segher Boessenkool wrote: >>> What does "uncached memory" even mean here? Literally it would be >>> I=1 memory (uncachEABLE memory), but more likely you want M=0 memory >>> here ("non-memory memory", "not well-behaved memory", MMIO often). >> >> Regular kernel memory vmapped with pgprot_noncached(). > > So, I=1 (and G=1). Caching inhibited and guarded. But M=1 (memory > coherence required) as with any other real memory :-) > >>> If memset() is expected to be used with M=0, you cannot do any serious >>> optimisations to it at all. If memset() is expected to be used with I=1 >>> it should use a separate code path for it, probably the caller should >>> make the distinction. >> >> DMA coherent memory which uses uncached memory for platforms that >> do not provide hardware dma coherence can end up just about anywhere >> in the kernel. We could use special routines for a few places in >> the DMA subsystem, but there might be plenty of others. > > Yeah. It will just be plenty slow, as we see here, that's what the > warning is for; but it works just fine :-) > > The memset() code itself could chech for the storage attributes, but > that is probably more expensive than just assuming the happy case. > Maybe someone could try it out though! Hmm, Ok! For what's worth I can at least test memset with dcbz+trap and what it was in 2015, without dcbz in the code path. How about that? I figured out of all the offenders (ethernet, crypto and sata). The sata/hard drive would be the most sensitive device to measure any performance difference. the MyBook Live already had an harddrive (Seagate ST380815AS (very old)) installed... so I went with that. I test with OpenWrt, since it has a fully working PowerPC images for the device, I can use initramfs (so HDD/SDD is idle) and provides a very bare minimum the hdparm -t "benchmark". (hdparm -t ... just reads for three seconds and tells you how much it read). the unmodified 6.6.47 kernel scored: | Timing buffered disk reads: 220 MB in 3.02 seconds = 72.93 MB/sec | Timing buffered disk reads: 222 MB in 3.02 seconds = 73.50 MB/sec | Timing buffered disk reads: 216 MB in 3.00 seconds = 71.94 MB/sec from what I can tell, each hdparm -t /dev/sda causes ~77000 fix_alignment traps. (/sys/devices/system/cpu/cpu0/cache/index0/coherency_line_size says it's 32 and type is obviously "Data". If I'm not mistaken this means ~2400KiB of emulated dcbz by the trap.) For the test, I added the "old" memset from and replaced 6.6.47's memset in dma_pool_alloc() with it now no WARNINGS are triggered and hdparm -t /dev/sda produces: | Timing buffered disk reads: 220 MB in 3.00 seconds = 73.32 MB/sec | Timing buffered disk reads: 218 MB in 3.02 seconds = 72.28 MB/sec | Timing buffered disk reads: 224 MB in 3.03 seconds = 74.02 MB/sec virtually no benefit?! Well, the HDD could be too slow. Let's try an old SSD: Samsung 840 Evo 120 GB. This one manages to read 1276 MB in 3.06 seconds = ~416 MB/sec in the same hdparm -t test on a reasonably modern PC when connected via a usb3<->sata adapter. unmodified 6.6.47 kernel: | Timing buffered disk reads: 356 MB in 3.00 seconds = 118.61 MB/sec | Timing buffered disk reads: 358 MB in 3.01 seconds = 119.12 MB/sec | Timing buffered disk reads: 358 MB in 3.01 seconds = 119.03 MB/sec modified 6.6.47 kernel: | Timing buffered disk reads: 380 MB in 3.01 seconds = 126.30 MB/sec | Timing buffered disk reads: 374 MB in 3.00 seconds = 124.61 MB/sec | Timing buffered disk reads: 382 MB in 3.02 seconds = 126.62 MB/sec Ok! There's something there. ~4%. Cheers, Christian