From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-il1-f174.google.com (mail-il1-f174.google.com [209.85.166.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2D9A63A1DA for ; Thu, 1 Aug 2024 16:27:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.166.174 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722529627; cv=none; b=ayU8jPg0xbfq+bAKSZMg7CUVnxcJJI7SL9fYe6gsNRHHET6z8GDAie5wufOKXola2ccK09ZD/RzaPoH/gpadQn9zJXegHpm6CCQQx6vc3daXFxZY9av5o7CoHXw23U3dXgYQYD8dPF0tlRt2qOcsnIIeZutlZnSVJWpjTcol6VU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722529627; c=relaxed/simple; bh=kmduuTslR1r6KFhm5dTaAgQsEJDGFeeYrbMrn4J3Cwc=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=AGKbCBRulRTROHBBaX9AAuNhAJo22oRV7l7EFbzvXzImxpIca2u/UofKNUpCHuE8JVWef4YzA3VFYppFZScBvYh66ACNud1Pn2ZC/s+mefAGbq7dcTrozLMEoZWe57empW4YkoPGJG95mVxg6TURja3Tn2xwKgy8haMxeFRm+UY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxfoundation.org; spf=pass smtp.mailfrom=linuxfoundation.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b=BnG9jhG7; arc=none smtp.client-ip=209.85.166.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linuxfoundation.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linuxfoundation.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b="BnG9jhG7" Received: by mail-il1-f174.google.com with SMTP id e9e14a558f8ab-39aeccc64e8so2394325ab.2 for ; Thu, 01 Aug 2024 09:27:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=google; t=1722529625; x=1723134425; darn=vger.kernel.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=3bS7017cZll4e+874qKazOOkVgc5570pVdFAP9FpAJU=; b=BnG9jhG7h/4vbW8oZN5sxsVnuHRlzi+CY+30SpDDReU/x9ziSQxzrViYNZEy2XF8Ji DxLTHWsMIfzIxIE5Yh7JNifCBnSQV0zaL7BEheNhZ290L3xWM2Q7YvIIcxJPKJjUuVZD JmBSr2t/tVKGR9awBbh26MH4jEHj+Pxgt293E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722529625; x=1723134425; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3bS7017cZll4e+874qKazOOkVgc5570pVdFAP9FpAJU=; b=j4/lho2q78e8sDEXNgNQjbwgPz1IaL+t+XDLzfJxOY4EDkZAhvH0/Pfw7/i/bRgTGZ OryTzNZG48GxTKvruNHsxx5uJ7z6i5X1D6r99lnrHnIupIiMXkrPBxOOZjyWcv2HH5/y 7T1l4heaSY9FXTIHyf9v5bTegd5ACh0soGJ+awpb9E1Xq8gFM/iR0MY3J1dbJZ0kbJ3Q 6GmvCbaMbTgIN29HVhFGtNMKUymJgKDvEN2DhlEhQZYs5ug/eZKpoUu3MOqHtJ+HV2Pf CGEBE4o0sn/VvI29gXFo+Yg8hOjMl6XORSr/y5CRaTz5xLgdxEUxZaQ4mJe3K36hP0mp Ontw== X-Forwarded-Encrypted: i=1; AJvYcCUEHC/UXcflvcRS1mxLt73uo7thLtDcgmY9CXr9tVkUnNk2XuwVQaTNvtSvTxEi1O8gq/gtS2Qn9Sz/gnI=@vger.kernel.org X-Gm-Message-State: AOJu0Yyy2mZJXXJUwLRfkznZzYADzx9ExdXBX1jl3XqPsvm+Y8NfkONX E2MwfsE+2Cz26rcQ2dVwAYDTLdM4cXizphrfZdaDCvcLe5/QIeKFNnq3ggDCBdQaz4BgUv2+Csg o X-Google-Smtp-Source: AGHT+IGErCOry7BajNV/z4Zkr4yfnMJio63Sv6XY9m5JQ814Wk5TMigZr92WLgckQyZqxSB0fNMfFg== X-Received: by 2002:a05:6e02:18ce:b0:381:c5f0:20d5 with SMTP id e9e14a558f8ab-39b1f772401mr5576285ab.0.1722529625303; Thu, 01 Aug 2024 09:27:05 -0700 (PDT) Received: from [192.168.1.128] ([38.175.170.29]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-39b20ae3537sm103715ab.61.2024.08.01.09.27.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 01 Aug 2024 09:27:04 -0700 (PDT) Message-ID: Date: Thu, 1 Aug 2024 10:27:04 -0600 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v2] selftests: openat2: don't print total number of tests and then skip To: Muhammad Usama Anjum , Shuah Khan Cc: Aleksa Sarai , kernel@collabora.com, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Shuah Khan References: <20240731133951.404933-1-usama.anjum@collabora.com> Content-Language: en-US From: Shuah Khan In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 8/1/24 02:42, Muhammad Usama Anjum wrote: > On 7/31/24 9:57 PM, Shuah Khan wrote: >> On 7/31/24 07:39, Muhammad Usama Anjum wrote: >>> Don't print that 88 sub-tests are going to be executed, but then skip. >>> This is against TAP compliance. Instead check pre-requisites first >>> before printing total number of tests. >> >> Does TAP clearly mention this? > Yes from https://testanything.org/tap-version-13-specification.html > > Skipping everything > This listing shows that the entire listing is a skip. No tests were run. > > TAP version 13 > 1..0 # skip because English-to-French translator isn't installed I don't see how this is applicable to the current scenario. The user needs to have root privilege to run the test. It is important to mention how many tests could have been run. As mentioned before, this information is important for users and testers. I would like to see this information in the output. > > We can see above that we need to print 1..0 and skip without printing the > total number of tests to be executed as they are going to be skipped. > >> >>> >>> Old non-tap compliant output: >>>    TAP version 13 >>>    1..88 >>>    ok 2 # SKIP all tests require euid == 0 >>>    # Planned tests != run tests (88 != 1)>>>    # Totals: pass:0 fail:0 xfail:0 xpass:0 skip:1 error:0 >>> >>> New and correct output: >>>    TAP version 13 >>>    1..0 # SKIP all tests require euid == 0 >> >> The problem is that this new output doesn't show how many tests >> are in this test suite that could be run. >> >> I am not use if this is better for communicating coverage information >> even if meets the TAP compliance. > I think the number of tests represents the number of planned tests. If we > don't plan to run X number of tests, we shouldn't print it. 88 tests are planned to be run except for the fact the first check failed. Planned tests could not be run because of user privileges. So these tests are all skips because of unmet dependencies. So the a good report would show that 88 tests could have been run. You can meet the specification and still make it work for us. When we adapt TAP 13 we didn't require 100% compliance. There are cases where you can comply and still provide how many test could be run. I think you are applying the spec strictly thereby removing useful information from the report. Can you tell me what would fail because of this "non-compliance"? thanks, -- Shuah