From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54589C11F68 for ; Tue, 29 Jun 2021 18:06:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3EF0861DE2 for ; Tue, 29 Jun 2021 18:06:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234884AbhF2SJJ (ORCPT ); Tue, 29 Jun 2021 14:09:09 -0400 Received: from youngberry.canonical.com ([91.189.89.112]:46157 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234894AbhF2SJB (ORCPT ); Tue, 29 Jun 2021 14:09:01 -0400 Received: from mail-ed1-f69.google.com ([209.85.208.69]) by youngberry.canonical.com with esmtps (TLS1.2) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.93) (envelope-from ) id 1lyI80-0005oa-Sf for linux-kernel@vger.kernel.org; Tue, 29 Jun 2021 18:06:32 +0000 Received: by mail-ed1-f69.google.com with SMTP id ee28-20020a056402291cb0290394a9a0bfaeso11838349edb.6 for ; Tue, 29 Jun 2021 11:06:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=1sVQNHAiS/W2i5o2hr99eStOUpYVcSh7ZDin7Mpc0cU=; b=YISjwQy2w8+DkavlwfDKEOVPxKVdbbABFRZ/AeT0TuJCZGOWY1mK6PSpgjsLCJnSi2 fi5r4JPaQt3zKSWCwk8fu+MOwT7dHACPJSevT9koaTnHz/VMo0xftbmknbhNdeFWSFqZ AfVLqqdRg0cplfmm3Gy1WaZa4Bpy8bPtJz/LiyizhHq4hT8ahpBZv7GLLJTHlYIL8npY Xa0PYZNU9l6KnSDWoeDz1wSemx74dMnHxHJaSlNqYivVD5OrHfqm5THCg07+yk9sFufZ GgOBqaEAg8G6fiMVaE4wDzzhduR9hKkNjo1GC7gR4jwx5yG5ABRIC/W0udwgDyKkMfnK utCw== X-Gm-Message-State: AOAM530fQeqSD+O4TvlM7kFxMR1KCEWS2Lz+9PfV7TrERoRo/sh7J6qM cURUFOKkGh8vTeauNxr5LnBi/4Qqf+NtMmwkSe2WBCUMsyQucz/YBFuyqXqfo+4FdFAgXT4a0QQ m1+aaMYwip2kdwlkXNEHaOLlFbnYBs1o6lYvPJxCreA== X-Received: by 2002:a50:935a:: with SMTP id n26mr42276814eda.8.1624989991235; Tue, 29 Jun 2021 11:06:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwTuOakoVILBoO7OTyCnoSp6AApNCYVL4pAvYGfXG5VRBZx+0rJbjTvqZHj1Uvtttw8JTlYRw== X-Received: by 2002:a50:935a:: with SMTP id n26mr42276796eda.8.1624989991101; Tue, 29 Jun 2021 11:06:31 -0700 (PDT) Received: from [192.168.1.115] (xdsl-188-155-177-222.adslplus.ch. [188.155.177.222]) by smtp.gmail.com with ESMTPSA id e13sm6969604ejl.98.2021.06.29.11.06.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 29 Jun 2021 11:06:30 -0700 (PDT) Subject: Re: [BUG] btrfs potential failure on 32 core LTP test (fallocate05) To: Josef Bacik , Chris Mason , David Sterba , linux-btrfs@vger.kernel.org, Linux Kernel Mailing List , "kernel-team@lists.ubuntu.com" , "ltp@lists.linux.it" , Qu Wenruo , Filipe Manana References: <124d7ead-6600-f369-7af1-a1bc27df135c@toxicpanda.com> <667133e5-44cb-8d95-c40a-12ac82f186f0@canonical.com> <0b6a502a-8db8-ef27-f48e-5001f351ef24@toxicpanda.com> From: Krzysztof Kozlowski Message-ID: <2576a472-1c99-889a-685c-a12bbfb08052@canonical.com> Date: Tue, 29 Jun 2021 20:06:30 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <0b6a502a-8db8-ef27-f48e-5001f351ef24@toxicpanda.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 29/06/2021 19:28, Josef Bacik wrote: > On 6/29/21 1:26 PM, Krzysztof Kozlowski wrote: >> On 29/06/2021 19:24, Josef Bacik wrote: >>> On 6/29/21 1:00 PM, Krzysztof Kozlowski wrote: >>>> Dear BTRFS folks, >>>> >>>> I am hitting a potential regression of btrfs, visible only with >>>> fallocate05 test from LTP (Linux Test Project) only on 32+ core Azure >>>> instances (x86_64). >>>> >>>> Tested: >>>> v5.8 (Ubuntu with our stable patches): PASS >>>> v5.11 (Ubuntu with our stable patches): FAIL >>>> v5.13 mainline: FAIL >>>> >>>> PASS means test passes on all instances >>>> FAIL means test passes on other instance types (e.g. 4 or 16 core) but >>>> fails on 32 and 64 core instances (did not test higher), >>>> e.g.: Standard_F32s_v2, Standard_F64s_v2, Standard_D32s_v3, >>>> Standard_E32s_v3 >>>> >>>> Reproduction steps: >>>> git clone https://github.com/linux-test-project/ltp.git >>>> cd ltp >>>> ./build.sh && make install -j8 >>>> cd ../ltp-install >>>> sudo ./runltp -f syscalls -s fallocate05 >>>> >>>> Failure output: >>>> tst_test.c:1379: TINFO: Testing on btrfs >>>> tst_test.c:888: TINFO: Formatting /dev/loop4 with btrfs opts='' extra opts='' >>>> tst_test.c:1311: TINFO: Timeout per run is 0h 05m 00s >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file0 size 21710183 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file1 size 8070086 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file2 size 3971177 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file3 size 36915315 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file4 size 70310993 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file5 size 4807935 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file6 size 90739786 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file7 size 76896492 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file8 size 72228649 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file9 size 36207821 >>>> tst_fill_fs.c:32: TINFO: Creating file mntpoint/file10 size 81483962 >>>> tst_fill_fs.c:59: TINFO: write(): ENOSPC (28) >>>> fallocate05.c:81: TPASS: write() wrote 65536 bytes >>>> fallocate05.c:102: TINFO: fallocate()d 0 extra blocks on full FS >>>> fallocate05.c:114: TPASS: fallocate() on full FS >>>> fallocate05.c:130: TPASS: fallocate(FALLOC_FL_PUNCH_HOLE | FALLOC_FL_KEEP_SIZE) >>>> fallocate05.c:134: TFAIL: write(): ENOSPC (28) >>>> >>>> Test code: >>>> https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/fallocate/fallocate05.c#L134 >>>> >>>> See also: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/1933112 >>>> >>>> Other FS tests succeed on that machines/kernels. Other file systems >>>> also pass - only btrfs fails. The issue was not bisected. Full test >>>> log attached. >>>> >>> >>> Also it looks like you're using a loop device, the instructions you gave me >>> aren't complete enough for me to reproduce. What is the actual setup you are >>> using? How big is your loop device? Is it a backing device? I had to do -b >>> to get the test to even start to run, but I've got a 2tib ssd, am I >>> supposed to be using something else? Thanks, >> >> The test takes care about loop device, nothing is needed from your side. >> Just run the test and wait till you see: >> "tst_test.c:1379: TINFO: Testing on btrfs" >> >> That's where the interesting part starts :) >> > > *cough* > # CONFIG_BLK_DEV_LOOP is not set > *cough* > > I think I found the problem, my bad, > Minor update - it's not only Azure's. AWS m5.8xlarge and m5.16xlarge (32 and 64 cores) fail similarly. I'll try later also QEMU machines with different amount of CPUs. Best regards, Krzysztof