* [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts
@ 2025-12-30 12:26 Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
0 siblings, 2 replies; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
To: Brendan Higgins, David Gow, Rae Moar
Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh,
Rae Moar
If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.
These patches where previously part of my KUnit UAPI series [0].
I split them out, as they are useful on their own and the KUnit UAPI
series does not actually have a hard dependency on them.
[0] https://lore.kernel.org/lkml/20250717-kunit-kselftests-v5-0-442b711cde2e@linutronix.de/
Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (2):
kunit: tool: Add test for nested test result reporting
kunit: tool: Don't overwrite test status based on subtest counts
tools/testing/kunit/kunit_parser.py | 3 +++
tools/testing/kunit/kunit_tool_test.py | 11 +++++++++++
.../kunit/test_data/test_is_test_passed-failure-nested.log | 10 ++++++++++
3 files changed, 24 insertions(+)
---
base-commit: a77b19b747c8a53ac16289685b0f77085a2ad553
change-id: 20251230-kunit-nested-failure-a3a56ec999ba
Best regards,
--
Thomas Weißschuh <thomas.weissschuh@linutronix.de>
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/2] kunit: tool: Add test for nested test result reporting 2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh @ 2025-12-30 12:26 ` Thomas Weißschuh 2026-01-05 8:55 ` David Gow 2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh 1 sibling, 1 reply; 5+ messages in thread From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw) To: Brendan Higgins, David Gow, Rae Moar Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh, Rae Moar Currently there is a lack of tests validating the result reporting from nested tests. Add one, it will also be used to validate upcoming changes to the nested test parsing. Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de> Reviewed-by: David Gow <davidgow@google.com> Reviewed-by: Rae Moar <rmoar@google.com> --- tools/testing/kunit/kunit_tool_test.py | 10 ++++++++++ .../kunit/test_data/test_is_test_passed-failure-nested.log | 7 +++++++ 2 files changed, 17 insertions(+) diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index bbba921e0eac..b74dc05fc2fe 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) self.assertEqual(result.counts.errors, 0) + def test_parse_failed_nested_tests_log(self): + nested_log = test_data_path('test_is_test_passed-failure-nested.log') + with open(nested_log) as file: + result = kunit_parser.parse_run_tests(file.readlines(), stdout) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) + self.assertEqual(result.counts.failed, 2) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status) + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status) + def test_no_header(self): empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log') with open(empty_log) as file: diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log new file mode 100644 index 000000000000..2e528da39ab5 --- /dev/null +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log @@ -0,0 +1,7 @@ +KTAP version 1 +1..2 +not ok 1 subtest 1 + KTAP version 1 + 1..1 + not ok 1 subsubtest 1 +not ok 2 subtest 2 -- 2.52.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 1/2] kunit: tool: Add test for nested test result reporting 2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh @ 2026-01-05 8:55 ` David Gow 0 siblings, 0 replies; 5+ messages in thread From: David Gow @ 2026-01-05 8:55 UTC (permalink / raw) To: Thomas Weißschuh Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev [-- Attachment #1: Type: text/plain, Size: 2753 bytes --] On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh <thomas.weissschuh@linutronix.de> wrote: > > Currently there is a lack of tests validating the result reporting from > nested tests. Add one, it will also be used to validate upcoming changes > to the nested test parsing. > > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de> > Reviewed-by: David Gow <davidgow@google.com> > Reviewed-by: Rae Moar <rmoar@google.com> > --- Thanks for splitting these out. This is still: Reviewed-by: David Gow <davidgow@google.com> (Note, though, that it introduces another call to test_data_path(), which conflicts with your pytest patches.) Cheers, -- David > tools/testing/kunit/kunit_tool_test.py | 10 ++++++++++ > .../kunit/test_data/test_is_test_passed-failure-nested.log | 7 +++++++ > 2 files changed, 17 insertions(+) > > diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py > index bbba921e0eac..b74dc05fc2fe 100755 > --- a/tools/testing/kunit/kunit_tool_test.py > +++ b/tools/testing/kunit/kunit_tool_test.py > @@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase): > self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) > self.assertEqual(result.counts.errors, 0) > > + def test_parse_failed_nested_tests_log(self): > + nested_log = test_data_path('test_is_test_passed-failure-nested.log') > + with open(nested_log) as file: > + result = kunit_parser.parse_run_tests(file.readlines(), stdout) > + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) > + self.assertEqual(result.counts.failed, 2) > + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status) > + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status) > + self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status) > + > def test_no_header(self): > empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log') > with open(empty_log) as file: > diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log > new file mode 100644 > index 000000000000..2e528da39ab5 > --- /dev/null > +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log > @@ -0,0 +1,7 @@ > +KTAP version 1 > +1..2 > +not ok 1 subtest 1 > + KTAP version 1 > + 1..1 > + not ok 1 subsubtest 1 > +not ok 2 subtest 2 > > -- > 2.52.0 > [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 5281 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts 2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh 2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh @ 2025-12-30 12:26 ` Thomas Weißschuh 2026-01-05 8:55 ` David Gow 1 sibling, 1 reply; 5+ messages in thread From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw) To: Brendan Higgins, David Gow, Rae Moar Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh If a subtest itself reports success, but the outer testcase fails, the whole testcase should be reported as a failure. However the status is recalculated based on the test counts, overwriting the outer test result. Synthesize a failed test in this case to make sure the failure is not swallowed. Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de> Reviewed-by: David Gow <davidgow@google.com> --- tools/testing/kunit/kunit_parser.py | 3 +++ tools/testing/kunit/kunit_tool_test.py | 1 + tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++ 3 files changed, 7 insertions(+) diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py index 333cd3a4a56b..5338489dcbe4 100644 --- a/tools/testing/kunit/kunit_parser.py +++ b/tools/testing/kunit/kunit_parser.py @@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None: elif test.counts.get_status() == TestStatus.TEST_CRASHED: test.status = TestStatus.TEST_CRASHED + if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS: + counts.add_status(status) + def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test: """ Finds next test to parse in LineStream, creates new Test object, diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py index b74dc05fc2fe..d5bbcb95ab6a 100755 --- a/tools/testing/kunit/kunit_tool_test.py +++ b/tools/testing/kunit/kunit_tool_test.py @@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase): self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) self.assertEqual(result.counts.failed, 2) self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status) + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status) self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status) self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status) diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log index 2e528da39ab5..5498dfd0b0db 100644 --- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log @@ -1,5 +1,8 @@ KTAP version 1 1..2 + KTAP version 1 + 1..1 + ok 1 test 1 not ok 1 subtest 1 KTAP version 1 1..1 -- 2.52.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts 2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh @ 2026-01-05 8:55 ` David Gow 0 siblings, 0 replies; 5+ messages in thread From: David Gow @ 2026-01-05 8:55 UTC (permalink / raw) To: Thomas Weißschuh Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev [-- Attachment #1: Type: text/plain, Size: 3092 bytes --] On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh <thomas.weissschuh@linutronix.de> wrote: > > If a subtest itself reports success, but the outer testcase fails, > the whole testcase should be reported as a failure. However the status > is recalculated based on the test counts, overwriting the outer test > result. Synthesize a failed test in this case to make sure the failure > is not swallowed. > > Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de> > Reviewed-by: David Gow <davidgow@google.com> > --- Thanks. Still looking good! Reviewed-by: David Gow <davidgow@google.com> Cheers, -- David > tools/testing/kunit/kunit_parser.py | 3 +++ > tools/testing/kunit/kunit_tool_test.py | 1 + > tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++ > 3 files changed, 7 insertions(+) > > diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py > index 333cd3a4a56b..5338489dcbe4 100644 > --- a/tools/testing/kunit/kunit_parser.py > +++ b/tools/testing/kunit/kunit_parser.py > @@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None: > elif test.counts.get_status() == TestStatus.TEST_CRASHED: > test.status = TestStatus.TEST_CRASHED > > + if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS: > + counts.add_status(status) > + > def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test: > """ > Finds next test to parse in LineStream, creates new Test object, > diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py > index b74dc05fc2fe..d5bbcb95ab6a 100755 > --- a/tools/testing/kunit/kunit_tool_test.py > +++ b/tools/testing/kunit/kunit_tool_test.py > @@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase): > self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status) > self.assertEqual(result.counts.failed, 2) > self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status) > + self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status) > self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status) > self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status) > > diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log > index 2e528da39ab5..5498dfd0b0db 100644 > --- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log > +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log > @@ -1,5 +1,8 @@ > KTAP version 1 > 1..2 > + KTAP version 1 > + 1..1 > + ok 1 test 1 > not ok 1 subtest 1 > KTAP version 1 > 1..1 > > -- > 2.52.0 > [-- Attachment #2: S/MIME Cryptographic Signature --] [-- Type: application/pkcs7-signature, Size: 5281 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-01-05 8:56 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh 2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh 2026-01-05 8:55 ` David Gow 2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh 2026-01-05 8:55 ` David Gow
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®