mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 00/12] kunit: Introduce UAPI testing framework
@ 2025-02-17 10:59 Thomas Weißschuh
  2025-02-17 10:59 ` [PATCH 01/12] kconfig: implement CONFIG_HEADERS_INSTALL for Usermode Linux Thomas Weißschuh
                   ` (12 more replies)
  0 siblings, 13 replies; 18+ messages in thread
From: Thomas Weißschuh @ 2025-02-17 10:59 UTC (permalink / raw)
  To: Masahiro Yamada, Nathan Chancellor, Nicolas Schier,
	Andrew Morton, Willy Tarreau, Thomas Weißschuh,
	Brendan Higgins, David Gow, Rae Moar, Shuah Khan,
	Jonathan Corbet
  Cc: linux-kbuild, linux-kernel, linux-kselftest, kunit-dev,
	linux-doc, Thomas Weißschuh

Currently testing of userspace and in-kernel API use two different
frameworks. kselftests for the userspace ones and Kunit for the
in-kernel ones. Besides their different scopes, both have different
strengths and limitations:

Kunit:
* Tests are normal kernel code.
* They use the regular kernel toolchain.
* They can be packaged and distributed as modules conveniently.

Kselftests:
* Tests are normal userspace code
* They need a userspace toolchain.
  A kernel cross toolchain is likely not enough.
* A fair amout of userland is required to run the tests,
  which means a full distro or handcrafted rootfs.
* There is no way to conveniently package and run kselftests with a
  given kernel image.
* The kselftests makefiles are not as powerful as regular kbuild.
  For example they are missing proper header dependency tracking or more
  complex compiler option modifications.

Therefore kunit is much easier to run against different kernel
configurations and architectures.
This series aims to combine kselftests and kunit, avoiding both their
limitations. It works by compiling the userspace kselftests as part of
the regular kernel build, embedding them into the kunit kernel or module
and executing them from there. If the kernel toolchain is not fit to
produce userspace because of a missing libc, the kernel's own nolibc can
be used instead.
The structured TAP output from the kselftest is integrated into the
kunit KTAP output transparently, the kunit parser can parse the combined
logs together.

Further room for improvements:
* Call each test in its completely dedicated namespace
* Handle additional test files besides the test executable through
  archives. CPIO, cramfs, etc.
* Compatibility with kselftest_harness.h (in progress)
* Expose the blobs in debugfs
* Provide some convience wrappers around compat userprogs
* Figure out a migration path/coexistence solution for
  kunit UAPI and tools/testing/selftests/

Output from the kunit example testcase, note the output of
"example_uapi_tests".

$ ./tools/testing/kunit/kunit.py run --kunitconfig lib/kunit example
...
Running tests with:
$ .kunit/linux kunit.filter_glob=example kunit.enable=1 mem=1G console=tty kunit_shutdown=halt
[11:53:53] ================== example (10 subtests) ===================
[11:53:53] [PASSED] example_simple_test
[11:53:53] [SKIPPED] example_skip_test
[11:53:53] [SKIPPED] example_mark_skipped_test
[11:53:53] [PASSED] example_all_expect_macros_test
[11:53:53] [PASSED] example_static_stub_test
[11:53:53] [PASSED] example_static_stub_using_fn_ptr_test
[11:53:53] [PASSED] example_priv_test
[11:53:53] =================== example_params_test  ===================
[11:53:53] [SKIPPED] example value 3
[11:53:53] [PASSED] example value 2
[11:53:53] [PASSED] example value 1
[11:53:53] [SKIPPED] example value 0
[11:53:53] =============== [PASSED] example_params_test ===============
[11:53:53] [PASSED] example_slow_test
[11:53:53] ======================= (4 subtests) =======================
[11:53:53] [PASSED] procfs
[11:53:53] [PASSED] userspace test 2
[11:53:53] [SKIPPED] userspace test 3: some reason
[11:53:53] [PASSED] userspace test 4
[11:53:53] ================ [PASSED] example_uapi_test ================
[11:53:53] ===================== [PASSED] example =====================
[11:53:53] ============================================================
[11:53:53] Testing complete. Ran 16 tests: passed: 11, skipped: 5
[11:53:53] Elapsed time: 67.543s total, 1.823s configuring, 65.655s building, 0.058s running

Based on v6.14-rc1 and the series
"tools/nolibc: compatibility with -Wmissing-prototypes" [0].
For compatibility with LLVM/clang another series is needed [1].

[0] https://lore.kernel.org/lkml/20250123-nolibc-prototype-v1-0-e1afc5c1999a@weissschuh.net/
[1] https://lore.kernel.org/lkml/20250213-kbuild-userprog-fixes-v1-0-f255fb477d98@linutronix.de/

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (12):
      kconfig: implement CONFIG_HEADERS_INSTALL for Usermode Linux
      kconfig: introduce CONFIG_ARCH_HAS_NOLIBC
      kbuild: userprogs: respect CONFIG_WERROR
      kbuild: userprogs: add nolibc support
      kbuild: introduce blob framework
      kunit: tool: Add test for nested test result reporting
      kunit: tool: Don't overwrite test status based on subtest counts
      kunit: tool: Parse skipped tests from kselftest.h
      kunit: Introduce UAPI testing framework
      kunit: uapi: Add example for UAPI tests
      kunit: uapi: Introduce preinit executable
      kunit: uapi: Validate usability of /proc

 Documentation/kbuild/makefiles.rst                 |  12 +
 Makefile                                           |   5 +-
 include/kunit/uapi.h                               |  17 ++
 include/linux/blob.h                               |  21 ++
 init/Kconfig                                       |   2 +
 lib/Kconfig.debug                                  |   1 -
 lib/kunit/Kconfig                                  |   9 +
 lib/kunit/Makefile                                 |  17 +-
 lib/kunit/kunit-example-test.c                     |  17 ++
 lib/kunit/kunit-uapi-example.c                     |  58 +++++
 lib/kunit/uapi-preinit.c                           |  61 +++++
 lib/kunit/uapi.c                                   | 250 +++++++++++++++++++++
 scripts/Makefile.blobs                             |  19 ++
 scripts/Makefile.build                             |   6 +
 scripts/Makefile.clean                             |   2 +-
 scripts/Makefile.userprogs                         |  18 +-
 scripts/blob-wrap.c                                |  27 +++
 tools/include/nolibc/Kconfig.nolibc                |  18 ++
 tools/testing/kunit/kunit_parser.py                |  13 +-
 tools/testing/kunit/kunit_tool_test.py             |   9 +
 .../test_is_test_passed-failure-nested.log         |  10 +
 .../test_data/test_is_test_passed-kselftest.log    |   3 +-
 22 files changed, 584 insertions(+), 11 deletions(-)
---
base-commit: 20e952894066214a80793404c9578d72ef89c5e0
change-id: 20241015-kunit-kselftests-56273bc40442

Best regards,
-- 
Thomas Weißschuh <thomas.weissschuh@linutronix.de>


^ permalink raw reply	[flat|nested] 18+ messages in thread

end of thread, other threads:[~2025-03-04 19:24 UTC | newest]

Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-02-17 10:59 [PATCH 00/12] kunit: Introduce UAPI testing framework Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 01/12] kconfig: implement CONFIG_HEADERS_INSTALL for Usermode Linux Thomas Weißschuh
2025-03-02 17:06   ` Masahiro Yamada
2025-03-04 19:23   ` Masahiro Yamada
2025-02-17 10:59 ` [PATCH 02/12] kconfig: introduce CONFIG_ARCH_HAS_NOLIBC Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 03/12] kbuild: userprogs: respect CONFIG_WERROR Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 04/12] kbuild: userprogs: add nolibc support Thomas Weißschuh
2025-02-22  9:44   ` Willy Tarreau
2025-02-17 10:59 ` [PATCH 05/12] kbuild: introduce blob framework Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 06/12] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 07/12] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 08/12] kunit: tool: Parse skipped tests from kselftest.h Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 09/12] kunit: Introduce UAPI testing framework Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 10/12] kunit: uapi: Add example for UAPI tests Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 11/12] kunit: uapi: Introduce preinit executable Thomas Weißschuh
2025-02-17 10:59 ` [PATCH 12/12] kunit: uapi: Validate usability of /proc Thomas Weißschuh
2025-02-18  8:20 ` [PATCH 00/12] kunit: Introduce UAPI testing framework David Gow
2025-02-18  8:57   ` Thomas Weißschuh

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®