Re: [PATCH 1/2] selftests: ftrace: Do not failure if there is unsupported tests

[Masami Hiramatsu <mhiramat@kernel.org>]

On Mon, 3 Jul 2017 09:15:43 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Mon,  3 Jul 2017 14:25:36 +0900
> Masami Hiramatsu <mhiramat@kernel.org> wrote:
> 
> > Do not return failure exit code (1) for unsupported testcases,
> > since it is expected for stable kernels.
> > 
> > Previously, ftracetest is expected to run only on current
> > release for avoiding regressions. However, nowadays we run
> > it on stable kernels. This means some test cases must return
> > unsupported result. In such case, we should NOT exit
> > ftracetest with error status for unsupported results so that
> > kselftest (upper tests wrapper) shows it passed correctly.
> 
> I wonder if we should change kselftest instead. There are case where we
> want to report "unsupported" as a failure. For instance, I have tests
> where I enable everything, and if a test returns "unsupported" then it
> is a failure for me.

OK, that's a possible usecase. 

> 
> Can we add an option in kselftest, or to ftracetest that decides if
> unsupported is a failure or not? Otherwise I can not ack this patch.

I would rather like to add an option to ftracetest instead of
kselftest, because whether the tested feature should be supported
or not is hard to decide from testing framework. It should be
checked by manual.

Thank you,

> 
> -- Steve
> 
> 
> > 
> > Note that we continue to treat unresolved results as failure,
> > if test writers would like to notice user that the test result
> > should be reviewed, they can use exit_unresolved.
> > 
> > Signed-off-by: Masami Hiramatsu <mhiramat@kernel.org>
> > ---
> >  tools/testing/selftests/ftrace/ftracetest |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/tools/testing/selftests/ftrace/ftracetest b/tools/testing/selftests/ftrace/ftracetest
> > index 14a03ea..290cd42 100755
> > --- a/tools/testing/selftests/ftrace/ftracetest
> > +++ b/tools/testing/selftests/ftrace/ftracetest
> > @@ -187,7 +187,7 @@ eval_result() { # sigval
> >      $UNSUPPORTED)
> >        prlog "	[UNSUPPORTED]"
> >        UNSUPPORTED_CASES="$UNSUPPORTED_CASES $CASENO"
> > -      return 1 # this is not a bug, but the result should be reported.
> > +      return 0 # this is not a bug.
> >      ;;
> >      $XFAIL)
> >        prlog "	[XFAIL]"
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>