Re: [LTP] [ANNOUNCE] The Linux Test Project has been Released for June 2007,

Re: [LTP] [ANNOUNCE] The Linux Test Project has been Released for June 2007,

[Randy Dunlap]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]

On Mon, 02 Jul 2007 12:28:50 +0530 Subrata Modak wrote:

>   :-) *Dear All*,
>  
> *Note(s) from the Maintainer: *
> * We have seen that the LTP mailing lists has become active for the last 
> couple of months. Thanks to all who has made this happen. We as an 
> integrated team can work hard to make this Project a resounding success. 
> Following are the different milestones/activities that we aim to 
> perform/achieve in the coming months:
> 1) Investigation into LTP Log(s) Library to make this more meaningful 
> and parsible. Kindly provide your inputs for the output that LTP 
> generates now and whether/how you would like it to be in future

I don't know if this qualifies as an input for the output, but
I use a script to parse the results file to compute a gross/overall
"SCORE" value.  Instead of looking at lines that contain
"termination_id" (e.g.:
duration=0 termination_type=exited termination_id=0 corefile=no
), I look at the test results output format messages and count
them (ignoring the INFO lines) and then compute a SCORE based on
all of those lines.  Obviously that's why I've been changing several
test cases to use the standard test results output format.

ltprun-summary.sh is attached.  It uses prtag2tag.pl, which is also
attached.  (I also do this summary SCORE for the Open Posix Test suite.)

Example summary + SCORE:

Using logfile=/cruc/runs/1789/test_output/ltp.log
build errors:  0 lines
PASS:          2225
WARNing:       1
BROKen:        1
RETiRed:       0
FAIL:          0
CONFig error:  11
SCORE.ltp:        99.5


> *Please also see the ChangeLog Below (June 2007):* *

so "[PATCH] float tests: use standard test results output"
(http://sourceforge.net/mailarchive/forum.php?thread_name=20070622214652.d137d055.rdunlap%40xenotime.net&forum_name=ltp-list)
will be added next month?

---
~Randy
*** Remember to use Documentation/SubmitChecklist when testing your code ***

[-- Attachment #2: ltprun-summary.sh --]
[-- Type: application/x-shellscript, Size: 1554 bytes --]

[-- Attachment #3: prtag2tag.pl --]
[-- Type: application/x-perl, Size: 744 bytes --]

[LTP] Validating exact PASS and FAIL scenarios for LTP

[Subrata Modak]

Hi All,

It may be a little high time for us to visit the way LTP defines itś
PASS and FAILURES. We know that each of LTP Test cases reports certain
keywords. Now, we need to define properly how each of these Keywords
needs to be interpreted in context to the addition information these
test-cases report, so that, we can arrive at saying that:
1) this information means that TEST has passed (and in such scenario the
test-case is bound to exit(0) to pan), and,
2) That set of information means that the TEST has failed (and in such
scenario the test-case is bound to exit(non-zero) to pan)

There are several reported incidents where there is ambiguity in judging
whether the test is a PASS or FAIL. Now we have come to a point where we
need to discuss and debate this out and say that:
1) These are the situations where the test should report exit(0) to pan
irrespective of what it has reported to logs through INFO:, and,
2) These are the situations where the test should immediately terminate
with exit(non-zero) to pan.

We also need to decide where these keywords like
PASS
WARN
BROK
FAIL
RETR
CONF
INFO
should be placed, and under what context ??

I hope all of you will contribute to this debate, where we finally seal
LTP´s fate as a:
1) Broken test-suite, or,
2) a glorious one.

--Regards--
Subrata

On Mon, 2007-07-02 at 08:50 -0700, Randy Dunlap wrote:
> On Mon, 02 Jul 2007 12:28:50 +0530 Subrata Modak wrote:
> 
> >   :-) *Dear All*,
> >  
> > *Note(s) from the Maintainer: *
> > generates now and whether/how you would like it to be in future
> 
> I don't know if this qualifies as an input for the output, but
>
> ltprun-summary.sh is attached.  It uses prtag2tag.pl, which is also
> attached.  (I also do this summary SCORE for the Open Posix Test suite.)
> 
> Example summary + SCORE:
> 
> Using logfile=/cruc/runs/1789/test_output/ltp.log
> build errors:  0 lines
> PASS:          2225
> WARNing:       1
> BROKen:        1
> RETiRed:       0
> FAIL:          0
> CONFig error:  11
> SCORE.ltp:        99.5
> 
> 
> > *Please also see the ChangeLog Below (June 2007):* *
> 
> so "[PATCH] float tests: use standard test results output"
> (http://sourceforge.net/mailarchive/forum.php?thread_name=20070622214652.d137d055.rdunlap%40xenotime.net&forum_name=ltp-list)
> will be added next month?
> 
> ---
> ~Randy
> *** Remember to use Documentation/SubmitChecklist when testing your code ***