[PATCH] selftests/lib.mk: run_tests= print summary at the end

[PATCH] selftests/lib.mk: run_tests= print summary at the end

[Matthieu Baerts (NGI0)]

When there are many tests and/or many log lines, it is difficult to
quickly find out how the different tests went when running:

    make -C tools/testing/selftests run_tests

This run_tests target executes 'run_many' from runner.sh, which relies
on ktap_helpers.sh. At the end of the execution, the KTAP_CNT_* counters
have been populated, and the total can then be simply printed with the
ktap_print_totals helper.

Now a summary is printed at the end, e.g. here with the two last lines:

    $ make -sC tools/testing/selftests TARGETS=(...) run_tests
    # timeout set to (...)
    # selftests: (...)
    (...)
    ok 1 selftests: (...)
    # timeout set to (...)
    # selftests: (...)
    (...)
    ok 2 selftests: (...) # SKIP
    # timeout set to (...)
    # selftests: (...)
    (...)
    not ok 3 selftests: (...)
    # 1 skipped test(s) detected.  Consider enabling relevant config options to improve coverage.
    # Totals: pass:1 fail:1 xfail:0 xpass:0 skip:1 error:0

This summary is printed, using '#' as prefix to be treated as a comment
in the TAP format. The summary is also printed when summary=1 is used to
redirect log lines in different files because it still looks useful when
there are many tests, not to have to check each line.

Note that this 'make' command will continue to exit without errors
(returned code is 0) when the tests can be executed. I guess it is
better not to change this behaviour to be able to continue detecting
kselftests failures.

Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
---
 tools/testing/selftests/lib.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index f02cc8a2e4ae..de15e63bc6b4 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -106,7 +106,8 @@ define RUN_TESTS
 	if [ "X$(summary)" != "X" ]; then       \
 		per_test_logging=1;		\
 	fi;                                     \
-	run_many $(1)
+	run_many $(1);				\
+	ktap_print_totals
 endef
 
 define INSTALL_INCLUDES

---
base-commit: fbb7ad31ab376c5101b2ac7205fad0344fd2de60
change-id: 20260625-sft-run-tests-print-totals-a18eea814bee

Best regards,
--  
Matthieu Baerts (NGI0) <matttbe@kernel.org>

Re: [PATCH] selftests/lib.mk: run_tests= print summary at the end

[Matthieu Baerts]

Hi Shuah,

On 25/06/2026 18:22, Matthieu Baerts (NGI0) wrote:
> When there are many tests and/or many log lines, it is difficult to
> quickly find out how the different tests went when running:
> 
>     make -C tools/testing/selftests run_tests
> 
> This run_tests target executes 'run_many' from runner.sh, which relies
> on ktap_helpers.sh. At the end of the execution, the KTAP_CNT_* counters
> have been populated, and the total can then be simply printed with the
> ktap_print_totals helper.
> 
> Now a summary is printed at the end, e.g. here with the two last lines:
> 
>     $ make -sC tools/testing/selftests TARGETS=(...) run_tests
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     ok 1 selftests: (...)
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     ok 2 selftests: (...) # SKIP
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     not ok 3 selftests: (...)
>     # 1 skipped test(s) detected.  Consider enabling relevant config options to improve coverage.
>     # Totals: pass:1 fail:1 xfail:0 xpass:0 skip:1 error:0
> 
> This summary is printed, using '#' as prefix to be treated as a comment
> in the TAP format. The summary is also printed when summary=1 is used to
> redirect log lines in different files because it still looks useful when
> there are many tests, not to have to check each line.
> 
> Note that this 'make' command will continue to exit without errors
> (returned code is 0) when the tests can be executed. I guess it is
> better not to change this behaviour to be able to continue detecting
> kselftests failures.
Gentle ping about this small addition? :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH] selftests/lib.mk: run_tests= print summary at the end

[Hangbin Liu]

On Thu, Jun 25, 2026 at 06:22:19PM +0200, Matthieu Baerts (NGI0) wrote:
> When there are many tests and/or many log lines, it is difficult to
> quickly find out how the different tests went when running:
> 
>     make -C tools/testing/selftests run_tests
> 
> This run_tests target executes 'run_many' from runner.sh, which relies
> on ktap_helpers.sh. At the end of the execution, the KTAP_CNT_* counters
> have been populated, and the total can then be simply printed with the
> ktap_print_totals helper.
> 
> Now a summary is printed at the end, e.g. here with the two last lines:
> 
>     $ make -sC tools/testing/selftests TARGETS=(...) run_tests
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     ok 1 selftests: (...)
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     ok 2 selftests: (...) # SKIP
>     # timeout set to (...)
>     # selftests: (...)
>     (...)
>     not ok 3 selftests: (...)
>     # 1 skipped test(s) detected.  Consider enabling relevant config options to improve coverage.
>     # Totals: pass:1 fail:1 xfail:0 xpass:0 skip:1 error:0
> 
> This summary is printed, using '#' as prefix to be treated as a comment
> in the TAP format. The summary is also printed when summary=1 is used to
> redirect log lines in different files because it still looks useful when
> there are many tests, not to have to check each line.
> 
> Note that this 'make' command will continue to exit without errors
> (returned code is 0) when the tests can be executed. I guess it is
> better not to change this behaviour to be able to continue detecting
> kselftests failures.
> 
> Signed-off-by: Matthieu Baerts (NGI0) <matttbe@kernel.org>
> ---
>  tools/testing/selftests/lib.mk | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
> index f02cc8a2e4ae..de15e63bc6b4 100644
> --- a/tools/testing/selftests/lib.mk
> +++ b/tools/testing/selftests/lib.mk
> @@ -106,7 +106,8 @@ define RUN_TESTS
>  	if [ "X$(summary)" != "X" ]; then       \
>  		per_test_logging=1;		\
>  	fi;                                     \
> -	run_many $(1)
> +	run_many $(1);				\
> +	ktap_print_totals
>  endef
>  
>  define INSTALL_INCLUDES

This helps get a clear summary for make run_tests.

Reviewed-by: Hangbin Liu <liuhangbin@kylinos.cn>