Re: [PATCH] selftests: Add information about TAP conformance in tests

[Muhammad Usama Anjum <usama.anjum@collabora.com>]

On 6/12/24 1:27 AM, Shuah Khan wrote:
> On 6/10/24 02:32, Muhammad Usama Anjum wrote:
>> Although "TAP" word is being used already in documentation, but it hasn't
>> been defined in informative way for developers that how to write TAP
>> conformant tests and what are the benefits. Write a short brief about it.
>>
>> Signed-off-by: Muhammad Usama Anjum <usama.anjum@collabora.com>
>> ---
>>   Documentation/dev-tools/kselftest.rst | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/Documentation/dev-tools/kselftest.rst
>> b/Documentation/dev-tools/kselftest.rst
>> index dcf634e411bd9..b579f491f3e97 100644
>> --- a/Documentation/dev-tools/kselftest.rst
>> +++ b/Documentation/dev-tools/kselftest.rst
>> @@ -228,6 +228,14 @@ In general, the rules for selftests are
>>    * Don't cause the top-level "make run_tests" to fail if your feature is
>>      unconfigured.
>>   + * The output of tests must conform to the TAP standard to ensure high
>> +   testing quality and to capture failures/errors with specific details.
>> +   The kselftest.h and kselftest_harness.h headers provide wrappers for
>> +   outputting test results such as pass, fail, or skip etc. These wrappers
>> +   should be used instead of reinventing the wheel or using raw printf and
> 
> When there are multiple tests supported by a test suite, it doesn't
> make sense to convert every single print statement. That is the reason
> why we have the run_tests wrapper that prints the summary in TAP format.
> 
> I would rephrase this to say "should be used for pass, fail, exit, and
> skip messages".
Makes sense. I'll update.

> 
> This rule is applicable to only the tests that are in the selftests Makefile
> and included in the default run.
> 
> There is no need to convert every single print statement in tests.
> 
>> +   exit statements. CI systems can easily parse TAP output messages to
>> +   detect test failures.
> 
> I would rather see people spending energy writing new tests and fixing
> warnings. TAP conversions need to be focused if they add value.
Some tests don't print pass/fail and just return status by exit value.
Adding pass/fail statements there really add value by exposing what went
wrong. So it can be reproduced in debug environment to fix it.

> 
> thanks,
> -- Shuah
> 

-- 
BR,
Muhammad Usama Anjum