mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
* [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts
@ 2025-12-30 12:26 Thomas Weißschuh
  2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
  2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
  0 siblings, 2 replies; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
  To: Brendan Higgins, David Gow, Rae Moar
  Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh,
	Rae Moar

If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.

These patches where previously part of my KUnit UAPI series [0].
I split them out, as they are useful on their own and the KUnit UAPI
series does not actually have a hard dependency on them.

[0] https://lore.kernel.org/lkml/20250717-kunit-kselftests-v5-0-442b711cde2e@linutronix.de/

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
---
Thomas Weißschuh (2):
      kunit: tool: Add test for nested test result reporting
      kunit: tool: Don't overwrite test status based on subtest counts

 tools/testing/kunit/kunit_parser.py                           |  3 +++
 tools/testing/kunit/kunit_tool_test.py                        | 11 +++++++++++
 .../kunit/test_data/test_is_test_passed-failure-nested.log    | 10 ++++++++++
 3 files changed, 24 insertions(+)
---
base-commit: a77b19b747c8a53ac16289685b0f77085a2ad553
change-id: 20251230-kunit-nested-failure-a3a56ec999ba

Best regards,
-- 
Thomas Weißschuh <thomas.weissschuh@linutronix.de>


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/2] kunit: tool: Add test for nested test result reporting
  2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
@ 2025-12-30 12:26 ` Thomas Weißschuh
  2026-01-05  8:55   ` David Gow
  2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
  1 sibling, 1 reply; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
  To: Brendan Higgins, David Gow, Rae Moar
  Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh,
	Rae Moar

Currently there is a lack of tests validating the result reporting from
nested tests. Add one, it will also be used to validate upcoming changes
to the nested test parsing.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: David Gow <davidgow@google.com>
Reviewed-by: Rae Moar <rmoar@google.com>
---
 tools/testing/kunit/kunit_tool_test.py                         | 10 ++++++++++
 .../kunit/test_data/test_is_test_passed-failure-nested.log     |  7 +++++++
 2 files changed, 17 insertions(+)

diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index bbba921e0eac..b74dc05fc2fe 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase):
 		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
 		self.assertEqual(result.counts.errors, 0)
 
+	def test_parse_failed_nested_tests_log(self):
+		nested_log = test_data_path('test_is_test_passed-failure-nested.log')
+		with open(nested_log) as file:
+			result = kunit_parser.parse_run_tests(file.readlines(), stdout)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
+		self.assertEqual(result.counts.failed, 2)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
+		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
+
 	def test_no_header(self):
 		empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
 		with open(empty_log) as file:
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
new file mode 100644
index 000000000000..2e528da39ab5
--- /dev/null
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -0,0 +1,7 @@
+KTAP version 1
+1..2
+not ok 1 subtest 1
+    KTAP version 1
+    1..1
+        not ok 1 subsubtest 1
+not ok 2 subtest 2

-- 
2.52.0


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts
  2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
  2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
@ 2025-12-30 12:26 ` Thomas Weißschuh
  2026-01-05  8:55   ` David Gow
  1 sibling, 1 reply; 5+ messages in thread
From: Thomas Weißschuh @ 2025-12-30 12:26 UTC (permalink / raw)
  To: Brendan Higgins, David Gow, Rae Moar
  Cc: linux-kernel, linux-kselftest, kunit-dev, Thomas Weißschuh

If a subtest itself reports success, but the outer testcase fails,
the whole testcase should be reported as a failure. However the status
is recalculated based on the test counts, overwriting the outer test
result. Synthesize a failed test in this case to make sure the failure
is not swallowed.

Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
Reviewed-by: David Gow <davidgow@google.com>
---
 tools/testing/kunit/kunit_parser.py                                  | 3 +++
 tools/testing/kunit/kunit_tool_test.py                               | 1 +
 tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
index 333cd3a4a56b..5338489dcbe4 100644
--- a/tools/testing/kunit/kunit_parser.py
+++ b/tools/testing/kunit/kunit_parser.py
@@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
 	elif test.counts.get_status() == TestStatus.TEST_CRASHED:
 		test.status = TestStatus.TEST_CRASHED
 
+	if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
+		counts.add_status(status)
+
 def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test:
 	"""
 	Finds next test to parse in LineStream, creates new Test object,
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index b74dc05fc2fe..d5bbcb95ab6a 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase):
 		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
 		self.assertEqual(result.counts.failed, 2)
 		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
+		self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status)
 		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
 		self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
 
diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
index 2e528da39ab5..5498dfd0b0db 100644
--- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
+++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
@@ -1,5 +1,8 @@
 KTAP version 1
 1..2
+    KTAP version 1
+    1..1
+        ok 1 test 1
 not ok 1 subtest 1
     KTAP version 1
     1..1

-- 
2.52.0


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH 1/2] kunit: tool: Add test for nested test result reporting
  2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
@ 2026-01-05  8:55   ` David Gow
  0 siblings, 0 replies; 5+ messages in thread
From: David Gow @ 2026-01-05  8:55 UTC (permalink / raw)
  To: Thomas Weißschuh
  Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev

[-- Attachment #1: Type: text/plain, Size: 2753 bytes --]

On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> Currently there is a lack of tests validating the result reporting from
> nested tests. Add one, it will also be used to validate upcoming changes
> to the nested test parsing.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Reviewed-by: David Gow <davidgow@google.com>
> Reviewed-by: Rae Moar <rmoar@google.com>
> ---

Thanks for splitting these out.

This is still:
Reviewed-by: David Gow <davidgow@google.com>

(Note, though, that it introduces another call to test_data_path(),
which conflicts with your pytest patches.)

Cheers,
-- David

>  tools/testing/kunit/kunit_tool_test.py                         | 10 ++++++++++
>  .../kunit/test_data/test_is_test_passed-failure-nested.log     |  7 +++++++
>  2 files changed, 17 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index bbba921e0eac..b74dc05fc2fe 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -165,6 +165,16 @@ class KUnitParserTest(unittest.TestCase):
>                 self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
>                 self.assertEqual(result.counts.errors, 0)
>
> +       def test_parse_failed_nested_tests_log(self):
> +               nested_log = test_data_path('test_is_test_passed-failure-nested.log')
> +               with open(nested_log) as file:
> +                       result = kunit_parser.parse_run_tests(file.readlines(), stdout)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
> +               self.assertEqual(result.counts.failed, 2)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
> +               self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
> +
>         def test_no_header(self):
>                 empty_log = test_data_path('test_is_test_passed-no_tests_run_no_header.log')
>                 with open(empty_log) as file:
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> new file mode 100644
> index 000000000000..2e528da39ab5
> --- /dev/null
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> @@ -0,0 +1,7 @@
> +KTAP version 1
> +1..2
> +not ok 1 subtest 1
> +    KTAP version 1
> +    1..1
> +        not ok 1 subsubtest 1
> +not ok 2 subtest 2
>
> --
> 2.52.0
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts
  2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
@ 2026-01-05  8:55   ` David Gow
  0 siblings, 0 replies; 5+ messages in thread
From: David Gow @ 2026-01-05  8:55 UTC (permalink / raw)
  To: Thomas Weißschuh
  Cc: Brendan Higgins, Rae Moar, linux-kernel, linux-kselftest, kunit-dev

[-- Attachment #1: Type: text/plain, Size: 3092 bytes --]

On Tue, 30 Dec 2025 at 20:26, Thomas Weißschuh
<thomas.weissschuh@linutronix.de> wrote:
>
> If a subtest itself reports success, but the outer testcase fails,
> the whole testcase should be reported as a failure. However the status
> is recalculated based on the test counts, overwriting the outer test
> result. Synthesize a failed test in this case to make sure the failure
> is not swallowed.
>
> Signed-off-by: Thomas Weißschuh <thomas.weissschuh@linutronix.de>
> Reviewed-by: David Gow <davidgow@google.com>
> ---

Thanks. Still looking good!

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David

>  tools/testing/kunit/kunit_parser.py                                  | 3 +++
>  tools/testing/kunit/kunit_tool_test.py                               | 1 +
>  tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log | 3 +++
>  3 files changed, 7 insertions(+)
>
> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py
> index 333cd3a4a56b..5338489dcbe4 100644
> --- a/tools/testing/kunit/kunit_parser.py
> +++ b/tools/testing/kunit/kunit_parser.py
> @@ -689,6 +689,9 @@ def bubble_up_test_results(test: Test) -> None:
>         elif test.counts.get_status() == TestStatus.TEST_CRASHED:
>                 test.status = TestStatus.TEST_CRASHED
>
> +       if status == TestStatus.FAILURE and test.counts.get_status() == TestStatus.SUCCESS:
> +               counts.add_status(status)
> +
>  def parse_test(lines: LineStream, expected_num: int, log: List[str], is_subtest: bool, printer: Printer) -> Test:
>         """
>         Finds next test to parse in LineStream, creates new Test object,
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index b74dc05fc2fe..d5bbcb95ab6a 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -172,6 +172,7 @@ class KUnitParserTest(unittest.TestCase):
>                 self.assertEqual(kunit_parser.TestStatus.FAILURE, result.status)
>                 self.assertEqual(result.counts.failed, 2)
>                 self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[0].status)
> +               self.assertEqual(kunit_parser.TestStatus.SUCCESS, result.subtests[0].subtests[0].status)
>                 self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].status)
>                 self.assertEqual(kunit_parser.TestStatus.FAILURE, result.subtests[1].subtests[0].status)
>
> diff --git a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> index 2e528da39ab5..5498dfd0b0db 100644
> --- a/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> +++ b/tools/testing/kunit/test_data/test_is_test_passed-failure-nested.log
> @@ -1,5 +1,8 @@
>  KTAP version 1
>  1..2
> +    KTAP version 1
> +    1..1
> +        ok 1 test 1
>  not ok 1 subtest 1
>      KTAP version 1
>      1..1
>
> --
> 2.52.0
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 5281 bytes --]

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-01-05  8:56 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-12-30 12:26 [PATCH 0/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2025-12-30 12:26 ` [PATCH 1/2] kunit: tool: Add test for nested test result reporting Thomas Weißschuh
2026-01-05  8:55   ` David Gow
2025-12-30 12:26 ` [PATCH 2/2] kunit: tool: Don't overwrite test status based on subtest counts Thomas Weißschuh
2026-01-05  8:55   ` David Gow

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®