[PATCH] selftests: Add information about TAP conformance in tests

[PATCH] selftests: Add information about TAP conformance in tests

[Muhammad Usama Anjum]

Although "TAP" word is being used already in documentation, but it hasn't
been defined in informative way for developers that how to write TAP
conformant tests and what are the benefits. Write a short brief about it.

Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
---
 Documentation/dev-tools/kselftest.rst | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index dcf634e411bd9..b579f491f3e97 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -228,6 +228,14 @@ In general, the rules for selftests are
  * Don't cause the top-level "make run_tests" to fail if your feature is
    unconfigured.
 
+ * The output of tests must conform to the TAP standard to ensure high
+   testing quality and to capture failures/errors with specific details.
+   The kselftest.h and kselftest_harness.h headers provide wrappers for
+   outputting test results such as pass, fail, or skip etc. These wrappers
+   should be used instead of reinventing the wheel or using raw printf and
+   exit statements. CI systems can easily parse TAP output messages to
+   detect test failures.
+
 Contributing new tests (details)
 ================================
 
-- 
2.39.2

Re: [PATCH] selftests: Add information about TAP conformance in tests

[Shuah Khan]

On 6/10/24 02:32, Muhammad Usama Anjum wrote:
> Although "TAP" word is being used already in documentation, but it hasn't
> been defined in informative way for developers that how to write TAP
> conformant tests and what are the benefits. Write a short brief about it.
> 
> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
> ---
>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index dcf634e411bd9..b579f491f3e97 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -228,6 +228,14 @@ In general, the rules for selftests are
>    * Don't cause the top-level "make run_tests" to fail if your feature is
>      unconfigured.
>   
> + * The output of tests must conform to the TAP standard to ensure high
> +   testing quality and to capture failures/errors with specific details.
> +   The kselftest.h and kselftest_harness.h headers provide wrappers for
> +   outputting test results such as pass, fail, or skip etc. These wrappers
> +   should be used instead of reinventing the wheel or using raw printf and

When there are multiple tests supported by a test suite, it doesn't
make sense to convert every single print statement. That is the reason
why we have the run_tests wrapper that prints the summary in TAP format.

I would rephrase this to say "should be used for pass, fail, exit, and
skip messages".

This rule is applicable to only the tests that are in the selftests Makefile
and included in the default run.

There is no need to convert every single print statement in tests.

> +   exit statements. CI systems can easily parse TAP output messages to
> +   detect test failures.

I would rather see people spending energy writing new tests and fixing
warnings. TAP conversions need to be focused if they add value.

thanks,
-- Shuah

Re: [PATCH] selftests: Add information about TAP conformance in tests

[Muhammad Usama Anjum]

On 6/12/24 1:27 AM, Shuah Khan wrote:
> On 6/10/24 02:32, Muhammad Usama Anjum wrote:
>> Although "TAP" word is being used already in documentation, but it hasn't
>> been defined in informative way for developers that how to write TAP
>> conformant tests and what are the benefits. Write a short brief about it.
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst
>> b/Documentation/dev-tools/kselftest.rst
>> index dcf634e411bd9..b579f491f3e97 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -228,6 +228,14 @@ In general, the rules for selftests are
>>    * Don't cause the top-level "make run_tests" to fail if your feature is
>>      unconfigured.
>>   + * The output of tests must conform to the TAP standard to ensure high
>> +   testing quality and to capture failures/errors with specific details.
>> +   The kselftest.h and kselftest_harness.h headers provide wrappers for
>> +   outputting test results such as pass, fail, or skip etc. These wrappers
>> +   should be used instead of reinventing the wheel or using raw printf and
> 
> When there are multiple tests supported by a test suite, it doesn't
> make sense to convert every single print statement. That is the reason
> why we have the run_tests wrapper that prints the summary in TAP format.
> 
> I would rephrase this to say "should be used for pass, fail, exit, and
> skip messages".
Makes sense. I'll update.

> 
> This rule is applicable to only the tests that are in the selftests Makefile
> and included in the default run.
> 
> There is no need to convert every single print statement in tests.
> 
>> +   exit statements. CI systems can easily parse TAP output messages to
>> +   detect test failures.
> 
> I would rather see people spending energy writing new tests and fixing
> warnings. TAP conversions need to be focused if they add value.
Some tests don't print pass/fail and just return status by exit value.
Adding pass/fail statements there really add value by exposing what went
wrong. So it can be reproduced in debug environment to fix it.

> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum