From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751790AbdKTQtE (ORCPT ); Mon, 20 Nov 2017 11:49:04 -0500 Received: from mail-io0-f195.google.com ([209.85.223.195]:36608 "EHLO mail-io0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751751AbdKTQtC (ORCPT ); Mon, 20 Nov 2017 11:49:02 -0500 X-Google-Smtp-Source: AGs4zMa/JowjORzedaJNWEmXJRisumoDfFv61vsY4PgzkFRytflA7FNfJHUNjYqPeJacXjTXicgDIw== Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: [LTP] Towards 4.14 LTS From: Tom Gall In-Reply-To: <20171120161029.GA2188@rei> Date: Mon, 20 Nov 2017 10:48:58 -0600 Cc: linux-kernel@vger.kernel.org, linux- stable , torvalds@linux-foundation.org, Greg Kroah-Hartman , linux-kselftest@vger.kernel.org, ltp@lists.linux.it, shuahkh@osg.samsung.com, Guenter Roeck Message-Id: References: <5E1947BC-9D22-42EB-BA29-96EA15A9C817@linaro.org> <20171120161029.GA2188@rei> To: Cyril Hrubis X-Mailer: Apple Mail (2.3445.4.7) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id vAKGn9wF020750 > On Nov 20, 2017, at 10:10 AM, Cyril Hrubis wrote: > > Hi! >> So why didn???t we report these? As mentioned we???ve been tossing out dodgy >> test cases to get to a clean baseline. We don???t need or want noise. >> >> For LTS, I want the system when it detects a failure to enable a quick >> bisect involving the affected test bucket. Given the nature of kernel >> bugs tho, there is that class of bug which only happens occasionally. > > From my experience debugging kernel bugs requires an actuall human > interaction and there is only certain level of automation that can be > achieved. Don't take me wrong, automatic bisection and other bells and > whistles are a nice to have, but at the end of the day you usually need > someone to reproduce/look at the problem, possibly check the source > code, report a bug, etc. Hence it does not make much sense to have an > automated system without dedicated engineers assigned to review the test > results. You are entirely right automation only gets so far. We have a few lines of defense that probably are worth a mention. 1) infra - sometimes results/runs need to be re-run for whatever reason. 2) triage - Crappy test case or something that is real? 3) kernel - bisecting etc We don’t have huge dedicated teams for each category but likewise each has a team. > -- > Cyril Hrubis > chrubis@suse.cz