From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756506AbdJJWks (ORCPT ); Tue, 10 Oct 2017 18:40:48 -0400 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:51846 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753333AbdJJWkr (ORCPT ); Tue, 10 Oct 2017 18:40:47 -0400 From: Nick Terrell To: "hpa@zytor.com" CC: Thomas Gleixner , Ingo Molnar , "linux-kernel@vger.kernel.org" , "x86@kernel.org" , Kernel Team , "Chris Mason" , Yann Collet , Rene Rebe Subject: Re: [PATCH 0/2] Add support for ZSTD-compressed kernel Thread-Topic: [PATCH 0/2] Add support for ZSTD-compressed kernel Thread-Index: AQHTQg6wMJEA6NKKREiW15iWYTq0QaLdoMUAgAAMloA= Date: Tue, 10 Oct 2017 22:40:13 +0000 Message-ID: <6BB37085-0223-4A61-81BF-18C450C210FD@fb.com> References: <20171010212244.1659577-1-terrelln@fb.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [2620:10d:c090:200::7:7f13] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;DM5PR15MB1210;20:jNNsbgkrDOmh5GZSSKfIs2tD9ou4ent2k/GtlgaZG8a7kZ0kK6kJbAg54inw1wfqK1uNC2FQEoWwajIgQRDVNl3IjgI6n71hQYYg0oEuJ0E35Z9ugvl2IQOB3fU1adaY+c4l7HXqp6qhcr5qudOggxftbSF+ezCKPsVlO4aMc84= x-ms-exchange-antispam-srfa-diagnostics: SSOS;SSOR; x-ms-office365-filtering-correlation-id: 18541d46-3e26-4bee-d54a-08d5102fdf34 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(2017030254152)(2017052603199)(201703131423075)(201703031133081)(201702281549075);SRVR:DM5PR15MB1210; x-ms-traffictypediagnostic: DM5PR15MB1210: x-exchange-antispam-report-test: UriScan:(67672495146484); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(93006095)(93001095)(10201501046)(3002001)(6041248)(20161123562025)(20161123560025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(20161123558100)(20161123555025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:DM5PR15MB1210;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:DM5PR15MB1210; x-forefront-prvs: 04569283F9 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(376002)(346002)(199003)(189002)(24454002)(377454003)(3280700002)(7736002)(8676002)(6512007)(4326008)(2906002)(14454004)(105586002)(2351001)(50986999)(189998001)(76176999)(53546010)(54356999)(106356001)(3660700001)(305945005)(6246003)(97736004)(25786009)(68736007)(81156014)(1730700003)(81166006)(53936002)(77096006)(36756003)(6436002)(6486002)(478600001)(101416001)(229853002)(82746002)(316002)(5640700003)(33656002)(2900100001)(83716003)(8936002)(54906003)(102836003)(99286003)(6506006)(2950100002)(6116002)(2501003)(86362001)(5660300001)(6916009);DIR:OUT;SFP:1102;SCL:1;SRVR:DM5PR15MB1210;H:DM5PR15MB1753.namprd15.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" Content-ID: <61187FF4FFA336429761DB64EF52316A@namprd15.prod.outlook.com> MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Oct 2017 22:40:13.5988 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR15MB1210 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-10-10_06:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by nfs id v9AMetj4018993 On 10/10/17, 2:56 PM, "hpa@zytor.com" wrote: >On October 10, 2017 2:22:42 PM PDT, Nick Terrell wrote: >>Hi, >> >>This patch set adds support for a ZSTD-compressed kernel and ramdisk >>images in the kernel boot process. It only integrates the support with >>x86, though the first patch is generic to all architectures. >> >>Zstandard requires slightly more memory during the kernel decompression >>on x86 (192 KB vs 64 KB), and the memory usage is independent of the >>window size. >> >>Zstandard requires memory proprortional to the window size used during >>compression for decompressing the ramdisk image, since streaming mode >>is >>used. Newer versions of zstd (1.3.2+) list the window size of a file >>with `zstd -lv '. The absolute maximum amount of memory required >>is just over 8 MB. >> >>Thanks, >>Nick Terrell >> >>Nick Terrell (2): >> lib: Add support for ZSTD-compressed kernel >> x86: Add support for ZSTD-compressed kernel >> >> Documentation/x86/boot.txt | 6 +- >> arch/x86/Kconfig | 1 + >> arch/x86/boot/compressed/Makefile | 5 +- >> arch/x86/boot/compressed/misc.c | 4 + >> arch/x86/boot/header.S | 8 +- >> arch/x86/include/asm/boot.h | 6 +- >> include/linux/decompress/unzstd.h | 26 +++ >> init/Kconfig | 14 +- >> lib/Kconfig | 4 + >> lib/Makefile | 1 + >> lib/decompress.c | 5 + >>lib/decompress_unzstd.c | 341 >>++++++++++++++++++++++++++++++++++++++ >> lib/xxhash.c | 21 ++- >> lib/zstd/decompress.c | 2 + >> lib/zstd/fse_decompress.c | 9 +- >> scripts/Makefile.lib | 15 ++ >> usr/Kconfig | 22 +++ >> 17 files changed, 465 insertions(+), 25 deletions(-) >> create mode 100644 include/linux/decompress/unzstd.h >> create mode 100644 lib/decompress_unzstd.c > >And, pray tell, what are the actual results? What is the trade-off of kernel size versus decompression performance versus the other algorithms that we already support? Adding algorithms for their own sake is a bad thing not a good thing. Sorry I neglected to include the benchmarks I've run so far. I've included them below, and will add them to the next version's cover letter. Comparing the command line tools on a kernel image that is 68970616 B large: | Algorithm | Compression Ratio | Decompression MB/s | |-----------|-------------------|--------------------| | zstd | 4.42 | 436.5 | | gzip | 3.72 | 134.1 | | xz | 4.83 | 53.1 | | lz4 | 3.18 | 1682.2 | | lzo | 3.36 | 389.6 | | bzip2 | 4.03 | 33.3 | This isn't actually benchmarks of the kernel decompression, but I expect the relative results to be the same. It compresses the second best and decompresses the second fastest. I know that this isn't a real benchmark of the kernel decompression. I still need to figure out how to time the kernel decompression. If you have any suggestions let me know. Otherwise, I'll get back to you when I've figured out how to run the benchmark. Best, Nick Terrell