From: John Ogness <john.ogness@linutronix.de>
To: Petr Mladek <pmladek@suse.com>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>,
Steven Rostedt <rostedt@goodmis.org>,
Thomas Gleixner <tglx@linutronix.de>,
linux-kernel@vger.kernel.org,
Francesco Dolcini <francesco@dolcini.it>,
kernel test robot <oliver.sang@intel.com>,
Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Subject: Re: [PATCH printk v3 02/14] printk: Adjust mapping for 32bit seq macros
Date: Mon, 15 Jan 2024 11:58:39 +0106 [thread overview]
Message-ID: <87a5p6g96w.fsf@jogness.linutronix.de> (raw)
In-Reply-To: <ZaGBlDTuQ0hYBpeJ@alley>
On 2024-01-12, Petr Mladek <pmladek@suse.com> wrote:
>> > Reported-by: Francesco Dolcini <francesco@dolcini.it>
>> > Reported-by: kernel test robot <oliver.sang@intel.com>
>> > Closes: https://lore.kernel.org/oe-lkp/202311171611.78d41dbe-oliver.sang@intel.com
>> > Reported-by: kernel test robot <oliver.sang@intel.com>
>> > Closes: https://lore.kernel.org/oe-lkp/202311161555.3ee16fc9-oliver.sang@intel.com
>> > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
>> > Signed-off-by: John Ogness <john.ogness@linutronix.de>
>>
>> Great catch! It must have been complicated to debug this.
>>
>> Reviewed-by: Petr Mladek <pmladek@suse.com>
>
> That said, I am a bit nervous that a bug like this might cause
> workqueue stall and panic() the kernel.
>
> At least, this is how I read
> https://lore.kernel.org/oe-lkp/202311171611.78d41dbe-oliver.sang@intel.com/
Note that the bug is reported for code that mainline will never
have. This patch "fixes" the problem before it is a problem. From the
perspective of mainline, the problem never existed. Perhaps it is
inappropriate to list these with the Closes tag.
> It looks like it caused some loop and refcout overlow or so.
> But I might be wrong.
>
> I would like to better understand this and check if we could prevent
> it somehow.
The code in question can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git/tree/kernel/printk/printk_ringbuffer.c?id=768c33033e41ad195a9304ebb183afb730d5ae98#n2031
The URL above points to prb_next_reserve_seq(), which is in an infinite
loop.
desc_last_finalized_seq() is returning a huge garbage value due to the
32bit sequence value 1 being incorrectly mapped to 0xffffffff00000001
(which is 18446744069414584321). When desc_read_finalized_seq() is
called (line 2082), it obviously returns -EINVAL because there is no
record with this huge sequence number.
The code interrupts -EINVAL to mean that the record has been
overwritten, and so it tries again with an updated last finalized
sequence number => infinite loop.
With this patch applied, the 32bit value 1 is correctly mapped to
sequence number 1 and all is OK.
The problem was that desc_last_finalized_seq() returned a sequence
number that was higher than the highest record. That was the bug. As
long as desc_last_finalized_seq() always returns a sequence number that
exists or used to exist, the retry loop is fine. And since the tail
record is always in the finalized state, there will always be a
finalized record available.
John
next prev parent reply other threads:[~2024-01-15 10:52 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-14 21:41 [PATCH printk v3 00/14] fix console flushing John Ogness
2023-12-14 21:41 ` [PATCH printk v3 01/14] printk: nbcon: Relocate 32bit seq macros John Ogness
2024-01-12 10:14 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 02/14] printk: Adjust mapping for " John Ogness
2023-12-15 9:55 ` Sebastian Andrzej Siewior
2023-12-15 10:10 ` John Ogness
2023-12-15 10:58 ` Sebastian Andrzej Siewior
2024-01-12 10:28 ` Petr Mladek
2024-01-12 18:14 ` Petr Mladek
2024-01-15 8:51 ` Sebastian Andrzej Siewior
2024-01-15 10:52 ` John Ogness [this message]
2024-01-15 16:17 ` Petr Mladek
2024-01-15 17:08 ` John Ogness
2023-12-14 21:41 ` [PATCH printk v3 03/14] printk: Use prb_first_seq() as base " John Ogness
2024-01-12 16:19 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 04/14] printk: ringbuffer: Do not skip non-finalized records with prb_next_seq() John Ogness
2024-01-12 18:05 ` Petr Mladek
2024-01-15 11:55 ` John Ogness
2024-01-15 17:00 ` Petr Mladek
2024-02-05 11:33 ` John Ogness
2024-02-06 17:27 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 05/14] printk: ringbuffer: Clarify special lpos values John Ogness
2024-01-30 13:12 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 06/14] printk: For @suppress_panic_printk check for other CPU in panic John Ogness
2023-12-14 21:41 ` [PATCH printk v3 07/14] printk: Add this_cpu_in_panic() John Ogness
2023-12-14 21:41 ` [PATCH printk v3 08/14] printk: ringbuffer: Cleanup reader terminology John Ogness
2024-01-30 14:36 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 09/14] printk: Wait for all reserved records with pr_flush() John Ogness
2024-01-31 11:36 ` Petr Mladek
2024-02-05 13:33 ` John Ogness
2024-02-07 9:20 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 10/14] printk: ringbuffer: Skip non-finalized records in panic John Ogness
2024-02-01 16:56 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 11/14] printk: ringbuffer: Consider committed as finalized " John Ogness
2024-02-01 18:00 ` Petr Mladek
2024-02-05 14:08 ` John Ogness
2024-02-07 10:11 ` Petr Mladek
2023-12-14 21:41 ` [PATCH printk v3 12/14] printk: Disable passing console lock owner completely during panic() John Ogness
2023-12-14 21:42 ` [PATCH printk v3 13/14] printk: Avoid non-panic CPUs writing to ringbuffer John Ogness
2024-02-02 9:26 ` Petr Mladek
2023-12-14 21:42 ` [PATCH printk v3 14/14] panic: Flush kernel log buffer at the end John Ogness
2024-02-02 9:30 ` Petr Mladek
2024-02-02 9:38 ` [PATCH printk v3 00/14] fix console flushing Petr Mladek
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87a5p6g96w.fsf@jogness.linutronix.de \
--to=john.ogness@linutronix.de \
--cc=bigeasy@linutronix.de \
--cc=francesco@dolcini.it \
--cc=linux-kernel@vger.kernel.org \
--cc=oliver.sang@intel.com \
--cc=pmladek@suse.com \
--cc=rostedt@goodmis.org \
--cc=senozhatsky@chromium.org \
--cc=tglx@linutronix.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
Powered by JetHome