From: "Chris Murphy" <lists@colorremedies.com>
To: "Justin Piszcz" <jpiszcz@lucidpixels.com>,
"Chaitanya Kulkarni" <chaitanyak@nvidia.com>
Cc: "linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
linux-kernel <linux-kernel@vger.kernel.org>,
"Btrfs BTRFS" <linux-btrfs@vger.kernel.org>
Subject: Re: BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 37868055...
Date: Tue, 11 Nov 2025 14:40:12 -0500 [thread overview]
Message-ID: <a383946c-60b4-49f2-a3d0-dda1a5b41117@app.fastmail.com> (raw)
In-Reply-To: <CAO9zADydZ=WrHbeREhWfetAupvzewM_A9Sz0RV8tY0h9CctoFA@mail.gmail.com>
On Tue, Nov 11, 2025, at 1:25 PM, Justin Piszcz wrote:
> Absolutely correct!! Luckily, I do have remote syslog enabled and
> captured the errors that were not logged to persistent storage, do the
> errors below point to an NVME firmware issue?
>
> "2025-11-10 01:42:51","notice","user","","machine1","[4043499.402974]
> BTRFS info (device nvme1n1p2): scrub: started on devid 2"
> "2025-11-10 01:42:51","notice","user","","machine1","[4043499.403000]
> BTRFS info (device nvme1n1p2): scrub: started on devid 1"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.683686]
> nvme nvme0: I/O tag 2 (a002) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.684742]
> nvme nvme0: I/O tag 3 (c003) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.685778]
> nvme nvme0: I/O tag 4 (1004) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.686802]
> nvme nvme0: I/O tag 5 (c005) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712274]
> nvme nvme0: I/O tag 6 (0006) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712483]
> nvme nvme0: I/O tag 7 (4007) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712698]
> nvme nvme0: I/O tag 8 (3008) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:22","notice","user","","machine1","[4043590.712906]
> nvme nvme0: I/O tag 9 (5009) opcode 0x2 (I/O Cmd) QID 2 timeout,
> aborting req_op:READ(0) size:131072"
> "2025-11-10 01:44:53","notice","user","","machine1","[4043621.419061]
> nvme nvme0: I/O tag 2 (a002) opcode 0x2 (I/O Cmd) QID 2 timeout, reset
> controller"
All of those read requests were dropped by the nvme controller and then the kernel nvme driver reset the conroller which appears to not work...
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.821452]
> nvme nvme0: Device not ready; aborting reset, CSTS=0x1"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850059]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850351]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850630]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.850903]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851173]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851440]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851705]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:14","notice","user","","machine1","[4043702.851971]
> nvme nvme0: Abort status: 0x371"
> "2025-11-10 01:46:34","notice","user","","machine1","[4043722.877036]
> nvme nvme0: Device not ready; aborting reset, CSTS=0x1"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.941289]
> pcieport 0000:00:1b.4: AER: Multiple Uncorrectable (Non-Fatal) error
> message received from 0000:06:00.0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.942254]
> nvme nvme0: Disabling device after reset failure: -19"
And now it's going to drop writes too because the device is disabled.
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969046]
> I/O error, dev nvme0n1, sector 86788096 op 0x1:(WRITE) flags 0x100000
> phys_seg 1 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969083]
> I/O error, dev nvme0n1, sector 464581888 op 0x0:(READ) flags 0x4000
> phys_seg 3 prio class 3"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969128]
> I/O error, dev nvme0n1, sector 45027984 op 0x1:(WRITE) flags 0x4000800
> phys_seg 1 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969127]
> I/O error, dev nvme0n1, sector 117882144 op 0x1:(WRITE) flags 0x1800
> phys_seg 8 prio class 2"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969146]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969146]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 2, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969163]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 4, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969160]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 3, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969167]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 5, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969167]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 6, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969179]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 7, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969179]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 8, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969184]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 9, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.969187]
> BTRFS error (device nvme1n1p2): bdev /dev/nvme0n1p2 errs: wr 10, rd 0,
> flush 0, corrupt 0, gen 0"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972280]
> BTRFS warning (device nvme1n1p2): lost super block write due to IO
> error on /dev/nvme0n1p2 (-5)"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972938]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844493824
> on dev /dev/nvme0n1p2 physical 237354287104"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972957]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844821504
> on dev /dev/nvme0n1p2 physical 237354614784"
> "2025-11-10 01:46:35","notice","user","","machine1","[4043722.972960]
> BTRFS error (device nvme1n1p2): fixed up error at logical 782844821504
> on dev /dev/nvme0n1p2 physical 237354614784"
> Got it, I was running 4B2QJXD7 firmware on both drives when this issue
> occurred. Recently, Samsung released a new NVME F/W update 7B2QJXD7,
> which they note "address(es) an intermittent non-recognition and blue
> screen issue." I've flashed the drives to 7B2QJXD7 and will see if
> this issue recurs.
That's probably the proper fix. The only other thing I can think of is there's a quirk for this drive that the kernel doesn't yet have, and therefore see if kernel 6.18-rc5 behaves better.
--
Chris Murphy
prev parent reply other threads:[~2025-11-11 19:40 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-11-10 15:05 Justin Piszcz
2025-11-11 16:56 ` Chris Murphy
2025-11-11 18:08 ` Chaitanya Kulkarni
2025-11-11 18:25 ` Justin Piszcz
2025-11-11 19:40 ` Chris Murphy [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a383946c-60b4-49f2-a3d0-dda1a5b41117@app.fastmail.com \
--to=lists@colorremedies.com \
--cc=chaitanyak@nvidia.com \
--cc=jpiszcz@lucidpixels.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®