* [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts
@ 2025-12-30 12:26 Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
0 siblings, 2 replies; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
To: Brendan Higgins, David Gow, Rae Moar
Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh,
Rae Moar
If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.
These patches where previously part of my KUnit UAPI series [0].
I split them out, as they are useful on their own and the KUnit UAPI
series does not actually have a hard dependency on them.
[0] https://lore.kernel.org/lkml/20250717-kunit-kselftests-v5-0-442b711cde2e@linutronix.de/
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (2):
kunit: tool: Add test for nested test result reporting
kunit: tool: Don't overwrite test status based on subtest counts
tools/testing/kunit/kunit_parser.py | 3 +++
tools/testing/kunit/kunit_tool_test.py | 11 +++++++++++
.../kunit/test_data/test_is_test_passed-failure-nested.log | 10 ++++++++++
3 files changed, 24 insertions(+)
---
base-commit: a77b19b747c8a53ac16289685b0f77085a2ad553
change-id: 20251230-kunit-nested-failure-a3a56ec999ba
Best regards,
--
Thomas Weißschuh <thomas.weissschuh@linutronix.de>
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 1/2] kunit: tool: Add test for nested test result reporting
2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
@ 2025-12-30 12:26 ` Thomas Weißschuh
2026-01-05 8:55 ` David Gow
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
1 sibling, 1 reply; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
To: Brendan Higgins, David Gow, Rae Moar
Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh,
Rae Moar
Currently there is a lack of tests validating the result reporting from
nested tests. Add one, it will also be used to validate upcoming changes
to the nested test parsing.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
---
tools/testing/kunit/kunit_tool_test.py | 10 ++++++++++
.../kunit/test_data/test_is_test_passed-failure-nested.log | 7 +++++++
2 files changed, 17 insertions(+)
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index bbba921e0eac..b74dc05fc2fe 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase):
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
self.assertEqual(result.counts.errors, 0)
+ def test_parse_failed_nested_tests_log(self):
+ nested_log = test_data_path('test_is_test_passed-failure-nested.log')
+ with open(nested_log) as file:
+ result = kunit_parser.parse_run_tests(file.readlines(), stdout)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
+ self.assertEqual(result.counts.failed, 2)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
+ self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
+
def test_no_header(self):
empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
with open(empty_log) as file:
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
new file mode 100644
index 000000000000..2e528da39ab5
--- /dev/null
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -0,0 +1,7 @@
+KTAP version 1
+1..2
+not ok 1 subtest 1
+ KTAP version 1
+ 1..1
+ not ok 1 subsubtest 1
+not ok 2 subtest 2
--
2.52.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts
2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
@ 2025-12-30 12:26 ` Thomas Weißschuh
2026-01-05 8:55 ` David Gow
1 sibling, 1 reply; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
To: Brendan Higgins, David Gow, Rae Moar
Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh
If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: David Gow <davidgow@google.com>
---
tools/testing/kunit/kunit_parser.py | 3 +++
tools/testing/kunit/kunit_tool_test.py | 1 +
tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
3 files changed, 7 insertions(+)
diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 333cd3a4a56b..5338489dcbe4 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
elif test.counts.get_status() == TestStatus.TEST_CRASHED:
test.status = TestStatus.TEST_CRASHED
+ if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
+ counts.add_status(status)
+
def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test:
"""
Finds next test to parse in LineStream, creates new Test object,
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index b74dc05fc2fe..d5bbcb95ab6a 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase):
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
self.assertEqual(result.counts.failed, 2)
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+ self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status)
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
index 2e528da39ab5..5498dfd0b0db 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -1,5 +1,8 @@
KTAP version 1
1..2
+ KTAP version 1
+ 1..1
+ ok 1 test 1
not ok 1 subtest 1
KTAP version 1
1..1
--
2.52.0
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] kunit: tool: Add test for nested test result reporting
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
@ 2026-01-05 8:55 ` David Gow
0 siblings, 0 replies; 5+ messages in thread
From: David Gow @ 2026-01-05 8:55 UTC (permalink / raw)
To: Thomas Weißschuh
Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev
[-- Attachment #1: Type: text/plain, Size: 2753 bytes --]
On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> Currently there is a lack of tests validating the result reporting from
> nested tests. Add one, it will also be used to validate upcoming changes
> to the nested test parsing.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Reviewed-by: David Gow <davidgow@google.com>
> Reviewed-by: Rae Moar <rmoar@google.com>
> ---
Thanks for splitting these out.
This is still:
Reviewed-by: David Gow <davidgow@google.com>
(Note, though, that it introduces another call to test_data_path(),
which conflicts with your pytest patches.)
Cheers,
-- David
> tools/testing/kunit/kunit_tool_test.py | 10 ++++++++++
> .../kunit/test_data/test_is_test_passed-failure-nested.log | 7 +++++++
> 2 files changed, 17 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index bbba921e0eac..b74dc05fc2fe 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase):
> self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
> self.assertEqual(result.counts.errors, 0)
>
> + def test_parse_failed_nested_tests_log(self):
> + nested_log = test_data_path('test_is_test_passed-failure-nested.log')
> + with open(nested_log) as file:
> + result = kunit_parser.parse_run_tests(file.readlines(), stdout)
> + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
> + self.assertEqual(result.counts.failed, 2)
> + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
> + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
> + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
> +
> def test_no_header(self):
> empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
> with open(empty_log) as file:
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> new file mode 100644
> index 000000000000..2e528da39ab5
> --- /dev/null
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> @@ -0,0 +1,7 @@
> +KTAP version 1
> +1..2
> +not ok 1 subtest 1
> + KTAP version 1
> + 1..1
> + not ok 1 subsubtest 1
> +not ok 2 subtest 2
>
> --
> 2.52.0
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
@ 2026-01-05 8:55 ` David Gow
0 siblings, 0 replies; 5+ messages in thread
From: David Gow @ 2026-01-05 8:55 UTC (permalink / raw)
To: Thomas Weißschuh
Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev
[-- Attachment #1: Type: text/plain, Size: 3092 bytes --]
On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> If a subtest itself reports success, but the outer testcase fails,
> the whole testcase should be reported as a failure. However the status
> is recalculated based on the test counts, overwriting the outer test
> result. Synthesize a failed test in this case to make sure the failure
> is not swallowed.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Reviewed-by: David Gow <davidgow@google.com>
> ---
Thanks. Still looking good!
Reviewed-by: David Gow <davidgow@google.com>
Cheers,
-- David
> tools/testing/kunit/kunit_parser.py | 3 +++
> tools/testing/kunit/kunit_tool_test.py | 1 +
> tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
> 3 files changed, 7 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 333cd3a4a56b..5338489dcbe4 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
> elif test.counts.get_status() == TestStatus.TEST_CRASHED:
> test.status = TestStatus.TEST_CRASHED
>
> + if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
> + counts.add_status(status)
> +
> def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test:
> """
> Finds next test to parse in LineStream, creates new Test object,
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index b74dc05fc2fe..d5bbcb95ab6a 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase):
> self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
> self.assertEqual(result.counts.failed, 2)
> self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
> + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status)
> self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
> self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
>
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> index 2e528da39ab5..5498dfd0b0db 100644
> --- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> @@ -1,5 +1,8 @@
> KTAP version 1
> 1..2
> + KTAP version 1
> + 1..1
> + ok 1 test 1
> not ok 1 subtest 1
> KTAP version 1
> 1..1
>
> --
> 2.52.0
>
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-01-05 8:56 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
2026-01-05 8:55 ` David Gow
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2026-01-05 8:55 ` David Gow
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®