From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932363AbdJKCCK (ORCPT ); Tue, 10 Oct 2017 22:02:10 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:43354 "EHLO mx0b-00082601.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756754AbdJKCCG (ORCPT ); Tue, 10 Oct 2017 22:02:06 -0400 From: Nick Terrell To: Adam Borowski CC: "hpa@zytor.com" , Thomas Gleixner , Ingo Molnar , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , "Kernel Team" , Chris Mason , Yann Collet , Rene Rebe Subject: Re: [PATCH 0/2] Add support for ZSTD-compressed kernel Thread-Topic: [PATCH 0/2] Add support for ZSTD-compressed kernel Thread-Index: AQHTQg6wMJEA6NKKREiW15iWYTq0QaLdoMUAgAAMloCAABiMgIAAH70A Date: Wed, 11 Oct 2017 02:01:41 +0000 Message-ID: References: <20171010212244.1659577-1-terrelln@fb.com> <6BB37085-0223-4A61-81BF-18C450C210FD@fb.com> <20171011000805.7b2nteb5hjbrfe7a@angband.pl> In-Reply-To: <20171011000805.7b2nteb5hjbrfe7a@angband.pl> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2620:10d:c090:200::7:7f13] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR15MB1209;20:ERH+bn0T3I2qwwJcdppe2wErhQS6CVoqys4Ni339vFcHhhBkPm5ob0dKMNMZiYI/CosyjncFS5bOlO+5+bBPLiJ93vBlm7vlWCrrZSR4faBJfXsog+4JXL6+AQdwEc448iZR1nC6vH7YDaFBedZ3/7Iw/Qh8SJi3oxyj63u171Q= x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-ms-office365-filtering-correlation-id: c1556a3d-4a1b-47ec-80dd-08d5104c046d x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075);SRVR:DM5PR15MB1209; x-ms-traffictypediagnostic: DM5PR15MB1209: x-exchange-antispam-report-test: UriScan:(67672495146484); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(2401047)(8121501046)(5005006)(3002001)(10201501046)(93006095)(93001095)(100000703101)(100105400095)(6041248)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123558100)(20161123564025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR15MB1209;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR15MB1209; x-forefront-prvs: 0457F11EAF x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(376002)(346002)(199003)(24454002)(377454003)(189002)(2906002)(3280700002)(54356999)(33656002)(189998001)(76176999)(106356001)(105586002)(50986999)(53546010)(6506006)(101416001)(86362001)(54906003)(2900100001)(81166006)(5660300001)(81156014)(6486002)(229853002)(8676002)(8936002)(77096006)(53936002)(14454004)(6436002)(4326008)(6246003)(6512007)(68736007)(25786009)(102836003)(93886005)(2950100002)(3660700001)(99286003)(82746002)(6916009)(305945005)(97736004)(7736002)(6116002)(83716003)(316002)(36756003)(478600001);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR15MB1209;H:DM5PR15MB1753.namprd15.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <1E9440AFDCA4634D80C037C35DEEFD29@namprd15.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 11 Oct 2017 02:01:41.8606 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1209 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-10-10_08:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v9B22MmU002123 On 10/10/17, 5:08 PM, "Adam Borowski" wrote: > On Tue, Oct 10, 2017 at 10:40:13PM +0000, Nick Terrell wrote: > > On 10/10/17, 2:56 PM, "hpa@zytor.com" wrote: > > >On October 10, 2017 2:22:42 PM PDT, Nick Terrell wrote: > > >>This patch set adds support for a ZSTD-compressed kernel and ramdisk > > >>images in the kernel boot process. It only integrates the support with > > >>x86, though the first patch is generic to all architectures. > > >> > > >>Zstandard requires slightly more memory during the kernel decompression > > >>on x86 (192 KB vs 64 KB), and the memory usage is independent of the > > >>window size. > > >> > > >>Zstandard requires memory proprortional to the window size used during > > >>compression for decompressing the ramdisk image, since streaming mode > > >>is > > >>used. Newer versions of zstd (1.3.2+) list the window size of a file > > >>with `zstd -lv '. The absolute maximum amount of memory required > > >>is just over 8 MB. > > > And, pray tell, what are the actual results? What is the trade-off of > > > kernel size versus decompression performance versus the other algorithms > > > that we already support? Adding algorithms for their own sake is a bad > > > thing not a good thing. > > > > Sorry I neglected to include the benchmarks I've run so far. I've included > > them below, and will add them to the next version's cover letter. > > > > Comparing the command line tools on a kernel image that is 68970616 B large: > > > > | Algorithm | Compression Ratio | Decompression MB/s | > > |-----------|-------------------|--------------------| > > | zstd | 4.42 | 436.5 | > > | gzip | 3.72 | 134.1 | > > | xz | 4.83 | 53.1 | > > | lz4 | 3.18 | 1682.2 | > > | lzo | 3.36 | 389.6 | > > | bzip2 | 4.03 | 33.3 | > Perhaps it'd be a good idea to cull some of bad algorithms? I don't know > the memory used by those, but envelope of the table you just shown suggests > using bzip2 and lzo is pointless. So is gzip, but it's widespread as the > default for initramfs producers, thus it'd be unsafe to kill it. I'm not sure there is a great use case for bzip2. It requires more memory than xz, compresses worse, and decompresses slower. lzo in the kernel might decompress a bit faster than zstd (looking back at the BtrFS benchmarks, it did). More importantly, it uses less memory than zstd. When decompressing the kernel zstd only needs 192 KB, but for initramfs, it will need more. Still, unless you really need 5% more compression, lz4 is probably a better option than lzo for speed. > > I know that this isn't a real benchmark of the kernel decompression. I > > still need to figure out how to time the kernel decompression. If you have > > any suggestions let me know. Otherwise, I'll get back to you when I've > > figured out how to run the benchmark. I've found a way to benchmark the kernel decompression time during boot with QEMU. I add timestamps to every line of the output. I also had to print 100 lines before the decompression starts to get consistent results. I've found that zstd is decompressing 2x slower than it should. I narrowed down the problem to ZSTD_wildcopy() and ZSTD_copy8() in lib/zstd/zstd_internal.h. ZSTD_wildcopy() calls memcpy(src, dst, 8) in a loop and doesn't handle the freestanding memcpy() well. Replacing it with __builtin_mcmpy(src, dst, 8) doubles the speed. I'm not an expert in freestanding gcc compilation, but I believe it is okay to call __builtin_memcpy() in freestanding mode, and gcc will either inline it, or add the right function call. The difference being that gcc will be able to apply its memcpy() analysis. I also see that arch/x86/boot/string.h defines memcpy() to __builtin_memcpy. Is it safe to directly use __builtin_memcpy() in lib/zstd/zstd_internal.h? If so, I'll submit a separate patch, and make sure to benchmark the existing use cases (BtrFS and SquashFS).