Re: [PATCH 4.9 00/67] 4.9.91-stable review

[Dan Rue <dan.rue@linaro.org>]

On Wed, Mar 28, 2018 at 06:21:04PM +0200, Greg Kroah-Hartman wrote:
> On Wed, Mar 28, 2018 at 10:50:35AM -0500, Dan Rue wrote:
> > On Wed, Mar 28, 2018 at 10:21:45AM +0200, Greg Kroah-Hartman wrote:
> > > On Tue, Mar 27, 2018 at 08:35:01PM -0500, Dan Rue wrote:
> > > > qemu_x86_64
> > > > * boot - pass: 21
> > > > * kselftest - skip: 28, pass: 52
> > > 
> > > Do you have a list of what you are skipping anywhere?  There was some
> > > x86 changes that I had to backport that I was worried about getting
> > > right here, are you running the x86 kselftests?
> > 
> > Yes we run the x86 selftests. Here's the full list of what ran and what was
> > skipped. Remember (I know you know, but for anyone else observing), we
> > run kselftest from 4.15.
> 
> That's good, as kselftest from 4.9 was a bit broken for x86, which is
> why I backported a bunch of patches for it for this release :)

I've noticed these backports. Should we expect to see selftests
supported in LTS kernels going forward? What is the current policy on
taking selftest backports?

> 
> >  breakpoint_test — SKIP
> >  efivarfs.sh — SKIP
> >  fsgsbase_64 — SKIP
> >  ftracetest — SKIP
> >  fw_fallback.sh — SKIP
> >  fw_filesystem.sh — SKIP
> >  gpio-mockup.sh — SKIP
> >  ldt_gdt_64 — SKIP
> >  main.sh — SKIP
> >  mem-on-off-test.sh — SKIP
> >  pstore_tests — SKIP
> >  reuseport_bpf — SKIP
> >  run.sh — SKIP
> >  run_vmtests — SKIP
> >  seccomp_bpf — SKIP
> >  sigreturn_64 — SKIP
> >  step_after_suspend_test — SKIP
> >  sync_test — SKIP
> >  sysctl.sh — SKIP
> >  test_align — SKIP
> >  test_dev_cgroup — SKIP
> >  test_kmod.sh — SKIP
> >  test_lpm_map — SKIP
> >  test_lru_map — SKIP
> >  test_maps — SKIP
> >  test_progs — SKIP
> >  test_tag — SKIP
> >  test_verifier — SKIP
> 
> Figuring out of any of these are from the tools/testing/selftests/x86/
> directory is a pain, that is the tests I was referring to here...

Agree. It would be nice if we preserved the directory name - I had the
same issue when trying to answer your question.

> 
> That being said, why are you skipping so many?  ldt_gdt_64 should be
> run, what fails with it?  Same for sigreturn_64, which I think means
> that only 2 tests from the x86 directory are being skipped, right?
> 
> Have you gone back and looked at any of those other tests to see why
> they are being skipped?

I'll try to keep this short - the tl;dr is that it's historical, and
something we're fixing now.

Originally, we just skipped everything that failed so that we could
focus on regressions. As new things failed, we have investigated them
and reported them upstream or fixed with patches. Over time, many of
these original failures have been resolved but the skipfiles may not
have been updated because...

Our skipfiles are per-branch and per-board, and we used to have one file
for each combination, resulting in dozens of files that ended up with
inconsistencies between them over time. So, we converted it to a
structured data format that is then parsed at runtime to generate a
skipfile for a given board/branch combination.

That work was just completed a couple weeks ago. Now that our kselftest
skipfile is a single structured data file[1], we need to go through it
and resolve all of the inconsistencies, remove the skips that have long
since been resolved, etc. This is work we plan to do in the next couple
weeks.

If you look through that file, you'll see quite a few things that are
wrong. When we're done auditing it, each skip should have a coherent
"reason" for skipping and an accurate branch/board list.

With regard to the specific tests you asked about:
- ldt_gdt_64: As of 4.9.89 this is still failing. I'll test again with
  the new RC.
- sigreturn_64: Passing on x86_64 (real hardware) and getting skipped on
  qemu x86_64. For some reason it fails intermittently on qemu in our
  environment.

Hope that helps,
Dan

[1] https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/kselftest/skipfile-lkft.yaml