kselftest backports?

kselftest backports?

[Tom Gall]

We've been running kselftests for ARM and x86 hardware in an effort to
detect regressions in various kernels including LTS and candidate
patches.

One general question we've had is what's the right thing to do when it
comes to running kselftest on older LTS kernels. Let's pick on 4.4 for
the purposes of this discussion tho obviously the example applies to
other versions.

1) Run current top of tree, kselftest on a 4.4 LTS?
2) Run kselftest from 4.4 on 4.4 LTS?

#1 gets latest greatest set of tests but obviously there can be
breakage because of how the kernel evolves over time.
#2 misses tests that are later developed but otherwise fine

Regardless of the right answer, some obvious questions but I'll ask anyway:
-) Shouldn't new additions to kselftest that work on older kernels be
backported and submitted on the stable list?
-) In the case where some test is kernel version specific for whatever
reason, I presume building in version detection into kselftest makes
sense?

Thanks.

-- 
Regards,
Tom

Director, Linaro Mobile Group
Linaro.org │ Open source software for ARM SoCs
irc: tgall_foo | @tom_gall

"Where's the kaboom!? There was supposed to be an earth-shattering
kaboom!" Marvin Martian

Re: kselftest backports?

[Greg KH]

On Thu, Jun 08, 2017 at 10:37:55AM -0500, Tom Gall wrote:
> We've been running kselftests for ARM and x86 hardware in an effort to
> detect regressions in various kernels including LTS and candidate
> patches.
> 
> One general question we've had is what's the right thing to do when it
> comes to running kselftest on older LTS kernels. Let's pick on 4.4 for
> the purposes of this discussion tho obviously the example applies to
> other versions.
> 
> 1) Run current top of tree, kselftest on a 4.4 LTS?
> 2) Run kselftest from 4.4 on 4.4 LTS?
> 
> #1 gets latest greatest set of tests but obviously there can be
> breakage because of how the kernel evolves over time.

Really?  What breaks?

> #2 misses tests that are later developed but otherwise fine
> 
> Regardless of the right answer, some obvious questions but I'll ask anyway:
> -) Shouldn't new additions to kselftest that work on older kernels be
> backported and submitted on the stable list?

No, that would be a new feature, and new features are not allowed in the
stable kernel releases, right?

And as the kernel/user api is "stable", there should never be any
problem with running new tests from newer kernel releases on older
kernel, right?  If so, I think that would be a bug and the test should
be fixed.  Right now we have a lot of tests that properly "degrade" if
the requested feature is not present, so I think things already work
properly today.

Do you know of any kselftests from 4.12-rc4 that are failing on 4.4
because of bugs in the tests?

> -) In the case where some test is kernel version specific for whatever
> reason, I presume building in version detection into kselftest makes
> sense?

No, it's a feature detection issue, look at how lots of tests already
degrade if a specific syscall is not present.  That should never be a
kernel release number detection as that would break horribly on things
like the "enterprise" Linux distro kernels who backport all sorts of new
features, and yet keep an old kernel version to make people feel warm
and fuzzy.

thanks,

greg k-h

Re: kselftest backports?

[Sumit Semwal]

Hi Greg,

On 8 June 2017 at 22:14, Greg KH <gregkh@linuxfoundation.org> wrote:
> On Thu, Jun 08, 2017 at 10:37:55AM -0500, Tom Gall wrote:
>> We've been running kselftests for ARM and x86 hardware in an effort to
>> detect regressions in various kernels including LTS and candidate
>> patches.
>>
>> One general question we've had is what's the right thing to do when it
>> comes to running kselftest on older LTS kernels. Let's pick on 4.4 for
>> the purposes of this discussion tho obviously the example applies to
>> other versions.
>>
>> 1) Run current top of tree, kselftest on a 4.4 LTS?
>> 2) Run kselftest from 4.4 on 4.4 LTS?
>>
>> #1 gets latest greatest set of tests but obviously there can be
>> breakage because of how the kernel evolves over time.
>
> Really?  What breaks?
>
So in what we've observed so far, testing a 4.4 kernel with say 4.10
kselftests, most of the failures are due to features getting added
later than the kernel being tested, but there are few that fail due to
behaviour changes in features.

To list a couple of them:
1. ftrace function glob filters:
- This fails on 4.4 because commit 60f1d5e3bac4: "ftrace: Support full
glob matching" got added in 4.9. Test case assumes it is present. (We
do have test case update pushed out for this upstream already)

2. seccomp test fails: [TRACE_syscall.skip_after_RET_TRACE FAIL]
- fails on 4.4 because the tested feature was added in linux-4.7 with
commit 58d0a862f573 ("seccomp: add tests for ptrace hole"), test case
assumes its presence.

3. sync framework tests fail:
- Commit 35538d7822e8 "dma-buf/sw_sync: de-stage SW_SYNC" got added in
4.8-rc2, corresponding kselftests got added in 4.9-rc1. (Again, we
have a test case update pushed out to skip testing based on feature
check).

4. sysctl tests failure (when CONFIG_PROC_SYSCTL_STRICT_WRITES isn't enabled)
- This is interesting because the test case checks for strict writes
for sysctl, but the default is non-strict-writes. So with kernels
built without explicit setting of strict_writes, test fails.
Curiously, 4.5+ kernels changed the default to be strict_writes, and
so the test passes there after. In this case, I checked with the
feature author and he feels its ok to backport the default-changing
patch to 4.4 as well, so I have it in my queue to propose for 4.4.y.

>> #2 misses tests that are later developed but otherwise fine
>>
>> Regardless of the right answer, some obvious questions but I'll ask anyway:
>> -) Shouldn't new additions to kselftest that work on older kernels be
>> backported and submitted on the stable list?
>
> No, that would be a new feature, and new features are not allowed in the
> stable kernel releases, right?
>
For the sake of argument: perhaps tests shouldn't be considered
features in this case? Because if we consider 'tests' as 'features',
then we couldn't be running new 'features' on older kernels anyways :)

> And as the kernel/user api is "stable", there should never be any
> problem with running new tests from newer kernel releases on older
> kernel, right?  If so, I think that would be a bug and the test should
> be fixed.  Right now we have a lot of tests that properly "degrade" if
> the requested feature is not present, so I think things already work
> properly today.
>
> Do you know of any kselftests from 4.12-rc4 that are failing on 4.4
> because of bugs in the tests?
>
Yes, a couple of examples above (though with 4.10 kselftests with 4.4.y)

>> -) In the case where some test is kernel version specific for whatever
>> reason, I presume building in version detection into kselftest makes
>> sense?
>
> No, it's a feature detection issue, look at how lots of tests already
> degrade if a specific syscall is not present.  That should never be a
> kernel release number detection as that would break horribly on things
> like the "enterprise" Linux distro kernels who backport all sorts of new
> features, and yet keep an old kernel version to make people feel warm
> and fuzzy.
>

I agree with you totally about the need to scrutinise the kselftests,
especially for failures due to missing features. We should probably
also look at making it a mandatory requirement for proposed tests to
degrade gracefully with older kernels before they're accepted in
kselftests?

How about the 'miscellaneous' failures though, especially ones
resulting from changes in feature behaviour evolution over kernels?

> thanks,
>
> greg k-h

Best,
Sumit.