mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Michal Wilczynski <m.wilczynski@samsung.com>
To: "S. Biplab Raut" <Biplab.Raut@amd.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Gao Xiang <xiang@kernel.org>,
	linux-erofs@lists.ozlabs.org,
	Yann Collet <yann.collet.73@gmail.com>,
	Nick Terrell <terrelln@meta.com>,
	Brendan Higgins <brendan.higgins@linux.dev>,
	David Gow <david@davidgow.net>, Rae Moar <raemoar63@gmail.com>,
	kunit-dev@googlegroups.com, linux-kselftest@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH] lib/lz4: add KUnit tests for compression and decompression
Date: Thu, 19 Nov 2026 20:43:45 +0100	[thread overview]
Message-ID: <e8235bd4-ad04-49d2-a5f6-f83463a8f8b6@samsung.com> (raw)
In-Reply-To: <20261118182343.1628236-1-Biplab.Raut@amd.com>



On 11/18/26 19:23, S. Biplab Raut wrote:
> lib/lz4 has no in-tree tests, and this series replaces both the
> compressor and the decompressor.  Add KUnit suites for the behaviour
> callers rely on.
> 
> lz4_decompress:
>  - round-trips through LZ4_decompress_safe() and
>    LZ4_decompress_safe_partial() with exact, oversized and partial
>    capacities;
>  - truncated and corrupted blocks never write past the destination,
>    through either entry point.  The input sits at the end of its
>    allocation, so KASAN also reports over-reads;
>  - an offset-0 match decodes the same whatever the buffer held before;
>  - literal and match lengths longer than the output are rejected.  On
>    32-bit, lengths whose extension bytes sum to SIZE_MAX are rejected
>    rather than wrapped;
>  - in-place decoding within LZ4_DECOMPRESS_INPLACE_MARGIN() leaves the
>    output intact.  A fixed block that corrupts with the old +32 margin
>    covers the vendored decoder's 32-byte wild copies, and a port of
>    upstream's worst-case test covers the 64-byte copies its decoder
>    makes since v1.11.0.
> 
> lz4_compress:
>  - LZ4_compress_default() fits LZ4_COMPRESSBOUND(), and with a smaller
>    buffer fails rather than overflowing;
>  - LZ4_compress_fast() across documented acceleration boundaries;
>  - LZ4_compress_destSize() output decodes to exactly the consumed
>    prefix;
>  - the dictionary paths of zram's lz4 and lz4hc backends;
>  - LZ4_compress_HC() across supported levels, including the kernel's
>    clamp boundary, with full and limited output.
> 
> Work memory is allocated at exactly LZ4_MEM_COMPRESS, LZ4HC_MEM_COMPRESS
> and LZ4_MEM_DECOMPRESS bytes, so KASAN reports an implementation that
> outgrows them.
> 
> Without this series, the 32-bit length case fails on i386 ("long match
> accepted: ret=31"): the old decoder wraps the match length to 18 bytes
> and accepts the block.
> 
> Signed-off-by: S. Biplab Raut <Biplab.Raut@amd.com>
> ---
> 
> Michal, this goes on top of your v2: it uses LZ4_MEM_DECOMPRESS and the
> opaque stream types, so please fold it into v3 if you find it useful.
> 
> The tests are deterministic regression tests.  Every randomized case
> uses a fixed seed, so each run checks the same inputs and any failure
> reproduces exactly.  The in-place suite has two fixed cases:
> 
>  - a block that corrupts with the old (compressedSize >> 8) + 32 margin
>    under a 32-byte fast loop, the case patch 12 fixes;
> 
>  - a port of upstream's decompress_safe_inPlace_worstCase test [1]:
>    fixed noise with a 65-byte match at offset 65, for every source size
>    from 300 to 4096.
> 
> On patch 12 and the +65 margin: since v1.11.0, upstream's decoder makes
> 64-byte wild copies on x86-64, aarch64 and ppc64, and upstream has raised
> its margin to (compressedSize >> 8) + 65 to match [1].  That fix is on
> the dev branch for v1.11.2; the released v1.11.0 and v1.11.1 headers
> still say +32.  Your v1.10.0 import uses the 32-byte loop, so +36 is
> right for this series.  A later re-sync to v1.11.0 or newer needs the
> kernel's margin raised to +65 in the same patch, whatever the upstream
> header at that tag says.
> 
> I checked that the worst-case test catches this, using a userspace
> build of the same test.  Against upstream dev's decoder, all 3797 sizes
> fail at +36 and none fail at +65.  Against your vendored v1.10.0
> decoder, none fail at +36.
> 
> The HC cases are built only when LZ4HC_COMPRESS is reachable.
> 
> With your v2 applied on v7.3-rc4, the suite passes on UML (17 passed,
> one 32-bit-only skip), on i386 (18 passed), and on x86_64 with KASAN and
> KASAN_VMALLOC (17 passed, the same skip).  A UML build without
> LZ4HC_COMPRESS passes 15 tests with the same skip.
> 
> [1] https://protect2.fireeye.com/v1/url?k=f02093a2-91ab868d-f02118ed-74fe485cbfe7-df7f89af9c73957c&q=1&e=79e95ae8-ccd9-4aaa-b235-c5f62451957b&u=https%3A%2F%2Fgithub.com%2Flz4%2Flz4%2Fpull%2F2104
> 
>  lib/Kconfig               |  15 +
>  lib/lz4/.kunitconfig      |   5 +
>  lib/lz4/Makefile          |   1 +
>  lib/lz4/tests/Makefile    |   2 +
>  lib/lz4/tests/lz4_kunit.c | 911 ++++++++++++++++++++++++++++++++++++++
>  5 files changed, 934 insertions(+)
>  create mode 100644 lib/lz4/.kunitconfig
>  create mode 100644 lib/lz4/tests/Makefile
>  create mode 100644 lib/lz4/tests/lz4_kunit.c
> 

Hi Biplab,

Thanks for the tests and for testing v2

v3 is coming in a couple of days. It vendors v1.11.1 instead of v1.10.0,
so it already has the 64-byte copies, and the in-place margin is
(compressedSize >> 8) + 68.

I ran your suite on v3. Everything passes except
lz4hc_compress_test_levels. That's because v3 no longer clamps HC levels
to 9: levels 10-12 need a bigger state, sized with the new
lz4hc_mem_compress(level). Allocating wrkmem per level with it fixes the
test.

If that's OK with you, I'll carry your patch in v3 with that small
change keeping you as the author. Since patches 10, 11 and 13 changed
a lot I'll drop your Tested-by - a re-test on v3 would be great.

Best regards,
-- 
Michal Wilczynski <m.wilczynski@samsung.com>

  reply	other threads:[~2026-11-19 19:43 UTC|newest]

Thread overview: 31+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CGME20261106144247eucas1p15bdd0704f32d1cb3b19f9d2296a0cbc3@eucas1p1.samsung.com>
2026-11-06 14:42 ` [PATCH v2 00/15] lib/lz4: stop forking upstream LZ4, vendor it instead Michal Wilczynski
     [not found]   ` <CGME20261106144249eucas1p1ca5f03b764c6305120da3e8b65f76ed1@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 01/15] lib/decompress_unlz4: use LZ4_decompress_safe_partial() before boot Michal Wilczynski
     [not found]   ` <CGME20261106144250eucas1p23dbd9566bcf265bb48728cedff315031@eucas1p2.samsung.com>
2026-11-06 14:42     ` [PATCH v2 02/15] lib/lz4: remove LZ4_decompress_fast() and its variants Michal Wilczynski
     [not found]   ` <CGME20261106144252eucas1p272c9e37d66ad385db070f941879aecb1@eucas1p2.samsung.com>
2026-11-06 14:42     ` [PATCH v2 03/15] zram: lz4hc: reset the dictionary stream once, not per page Michal Wilczynski
     [not found]   ` <CGME20261106144254eucas1p12df3a03a5f8e74ca4a04931ae80812bf@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 04/15] lib/lz4: import upstream LZ4 sources verbatim Michal Wilczynski
     [not found]   ` <CGME20261106144256eucas1p1beb947272dc94f916c8c8131722a480d@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 05/15] lib/lz4: backport upstream's -Wmissing-prototypes fix Michal Wilczynski
     [not found]   ` <CGME20261106144258eucas1p1f59ead3e337482e074bfb26f4553e70d@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 06/15] lib/lz4: backport upstream's aarch64 fast decode loop enablement Michal Wilczynski
     [not found]   ` <CGME20261106144259eucas1p27d26efbc44beac102c03ec07ebed4310@eucas1p2.samsung.com>
2026-11-06 14:42     ` [PATCH v2 07/15] lib/lz4: backport upstream's match length limit fix Michal Wilczynski
     [not found]   ` <CGME20261106144301eucas1p1882c2f6ac262ef7c10c3e0cabe333a6b@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 08/15] lib/lz4: add the build environment for the vendored sources Michal Wilczynski
     [not found]   ` <CGME20261106144303eucas1p10d145c55d4e168ac6d2d12758125a532@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 09/15] arch: boot: put the LZ4 freestanding headers on the decompressor path Michal Wilczynski
     [not found]   ` <CGME20261106144305eucas1p17b3b27d403dc7d13b85cdd9a4bea83b9@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 10/15] lib/lz4: switch the compressor to the vendored sources Michal Wilczynski
2026-11-07  1:40       ` kernel test robot
2026-11-10  5:36       ` Sergey Senozhatsky
2026-11-12 12:02         ` Michal Wilczynski
2026-11-18 18:35       ` S. Biplab Raut
     [not found]   ` <CGME20261106144306eucas1p17d110f5e01991516acd49252f81845b2@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 11/15] lib/lz4: switch the HC " Michal Wilczynski
2026-11-18 18:35       ` S. Biplab Raut
     [not found]   ` <CGME20261106144309eucas1p2a9bfcc1cf81779d473fb073e7d9139b7@eucas1p2.samsung.com>
2026-11-06 14:42     ` [PATCH v2 12/15] lib/lz4: widen LZ4_DECOMPRESS_INPLACE_MARGIN for the fast decode loop Michal Wilczynski
2026-11-11 22:02       ` Gao Xiang
2026-11-12 12:40         ` Michal Wilczynski
2026-11-18 18:35       ` S. Biplab Raut
     [not found]   ` <CGME20261106144310eucas1p1d60fe5c2e0170c49f38a3142bb1d8f94@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 13/15] lib/lz4: switch the decompressor to the vendored sources Michal Wilczynski
2026-11-10  5:39       ` Sergey Senozhatsky
2026-11-12 12:06         ` Michal Wilczynski
2026-11-18 18:35       ` S. Biplab Raut
     [not found]   ` <CGME20261106144312eucas1p26109d26cecef034bf7734283cb04b211@eucas1p2.samsung.com>
2026-11-06 14:42     ` [PATCH v2 14/15] lib/lz4: replace lz4_kernel_api.h's prototypes with <linux/lz4.h> Michal Wilczynski
     [not found]   ` <CGME20261106144313eucas1p18ebd11c40d0e1e68730aac1800b2502f@eucas1p1.samsung.com>
2026-11-06 14:42     ` [PATCH v2 15/15] MAINTAINERS: add an entry for the LZ4 compression library Michal Wilczynski
2026-11-12 11:25   ` [PATCH v2 00/15] lib/lz4: stop forking upstream LZ4, vendor it instead Michal Wilczynski
2026-11-18 18:23   ` [PATCH] lib/lz4: add KUnit tests for compression and decompression S. Biplab Raut
2026-11-19 19:43     ` Michal Wilczynski [this message]
2026-11-20 12:23       ` S. Biplab Raut

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=e8235bd4-ad04-49d2-a5f6-f83463a8f8b6@samsung.com \
    --to=m.wilczynski@samsung.com \
    --cc=Biplab.Raut@amd.com \
    --cc=akpm@linux-foundation.org \
    --cc=brendan.higgins@linux.dev \
    --cc=david@davidgow.net \
    --cc=kunit-dev@googlegroups.com \
    --cc=linux-erofs@lists.ozlabs.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=raemoar63@gmail.com \
    --cc=terrelln@meta.com \
    --cc=xiang@kernel.org \
    --cc=yann.collet.73@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®