From: Song Liu <songliubraving@fb.com>
To: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: Rik van Riel <riel@fb.com>, "song@kernel.org" <song@kernel.org>,
"joe.lawrence@redhat.com" <joe.lawrence@redhat.com>,
"peterz@infradead.org" <peterz@infradead.org>,
"mingo@redhat.com" <mingo@redhat.com>,
"vincent.guittot@linaro.org" <vincent.guittot@linaro.org>,
"pmladek@suse.com" <pmladek@suse.com>,
"live-patching@vger.kernel.org" <live-patching@vger.kernel.org>,
Kernel Team <Kernel-team@fb.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"jpoimboe@redhat.com" <jpoimboe@redhat.com>
Subject: Re: [RFC] sched,livepatch: call klp_try_switch_task in __cond_resched
Date: Tue, 10 May 2022 23:57:04 +0000 [thread overview]
Message-ID: <D298A3F1-43A5-4FD5-B198-906364BF4B79@fb.com> (raw)
In-Reply-To: <20220510230402.e5ymkwt45sg7bd35@treble>
> On May 10, 2022, at 4:04 PM, Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Tue, May 10, 2022 at 07:45:49PM +0000, Song Liu wrote:
>>>> A KLP transition preempt notifier would help those
>>>> kernel threads transition to the new KLP version at
>>>> any time they reschedule.
>>>
>>> ... unless cond_resched() is a no-op due to CONFIG_PREEMPT?
>>
>> Based on my understanding (and a few other folks we chatted with),
>> a kernel thread can legally run for extended time, as long as it
>> calls cond_resched() at a reasonable frequency. Therefore, I
>> think we should be able to patch such thread easily, unless it
>> calls cond_resched() with being-patched function in the stack,
>> of course.
>
> But again, with CONFIG_PREEMPT, that doesn't work.
>
>> OTOH, Petr's mindset of allowing many minutes for the patch
>> transition is new to me. I need to think more about it.
>> Josh, what’s you opinion on this? IIUC, kpatch is designed to
>> only wait up to 60 seconds (no option to overwrite the time).
>
> I wouldn't be necessarily opposed to changing the kpatch timeout to
> something bigger, or eliminating it altogether in favor of a WARN()
> after x minutes.
>
>>>> How much it will help is hard to predict, but I should
>>>> be able to get results from a fairly large sample size
>>>> of systems within a few weeks :)
>>>
>>> As Peter said, keep in mind that we will need to fix other cases beyond
>>> Facebook, i.e., CONFIG_PREEMPT combined with non-x86 arches which don't
>>> have ORC so they can't reliably unwind from an IRQ.
>>
>> I think livepatch transition may fail in different cases, and we
>> don't need to address all of them in one shoot. Fixing some cases
>> is an improvement as long as we don't slow down other cases. I
>> understand that adding tiny overhead to __cond_resched() may end
>> up as a visible regression. But maybe adding it to
>> preempt_schedule_common() is light enough?
>>
>> Did I miss/misunderstand something?
>
> If it's a real bug, we should fix it everywhere, not just for Facebook.
> Otherwise CONFIG_PREEMPT and/or non-x86 arches become second-class
> citizens.
I think "is it a real bug?" is the top question for me. So maybe we
should take a step back.
The behavior we see is: A busy kernel thread blocks klp transition
for more than a minute. But the transition eventually succeeded after
< 10 retries on most systems. The kernel thread is well-behaved, as
it calls cond_resched() at a reasonable frequency, so this is not a
deadlock.
If I understand Petr correctly, this behavior is expected, and thus
is not a bug or issue for the livepatch subsystem. This is different
to our original expectation, but if this is what we agree on, we
will look into ways to incorporate long wait time for patch
transition in our automations.
OTOH, if we think this is a suboptimal behavior (not necessarily a
bug, IIUC), we should consider improve it. I personally don’t think
shipping an improvement to one configuration makes other configurations
second-class citizens. And, it is possible PREEMPT kernel does NOT
even have such suboptimal behavior, right? (I really don't know.)
So, if we come back to the same question: is this a bug (or a suboptimal
behavior that worth fixing)? If so, we are open to any solution that
would also help PREEMPT and/or non-x86 arches.
Lastly, maybe a really naive question: does the following also helps
PREEMPT=y configurations?
Thanks,
Song
diff --git c/kernel/sched/core.c w/kernel/sched/core.c
index a7302b7b65f2..cc9b1c9c02ba 100644
--- c/kernel/sched/core.c
+++ w/kernel/sched/core.c
@@ -5226,6 +5226,9 @@ void __sched schedule_preempt_disabled(void)
static void __sched notrace preempt_schedule_common(void)
{
+ if (unlikely(klp_patch_pending(current)))
+ klp_try_switch_task(current);
+
do {
/*
* Because the function tracer can trace preempt_count_sub()
next prev parent reply other threads:[~2022-05-10 23:57 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-07 17:46 Song Liu
2022-05-07 18:26 ` Rik van Riel
2022-05-07 19:04 ` Song Liu
2022-05-07 19:18 ` Rik van Riel
2022-05-08 20:41 ` Peter Zijlstra
2022-05-09 1:07 ` Rik van Riel
2022-05-09 7:04 ` Peter Zijlstra
2022-05-09 8:06 ` Song Liu
2022-05-09 9:38 ` Peter Zijlstra
2022-05-09 14:13 ` Rik van Riel
2022-05-09 15:22 ` Petr Mladek
2022-05-09 15:07 ` Petr Mladek
2022-05-09 16:22 ` Song Liu
2022-05-10 7:56 ` Petr Mladek
2022-05-10 13:33 ` Rik van Riel
2022-05-10 15:44 ` Petr Mladek
2022-05-10 16:07 ` Rik van Riel
2022-05-10 16:52 ` Josh Poimboeuf
2022-05-10 18:07 ` Rik van Riel
2022-05-10 18:42 ` Josh Poimboeuf
2022-05-10 19:45 ` Song Liu
2022-05-10 23:04 ` Josh Poimboeuf
2022-05-10 23:57 ` Song Liu [this message]
2022-05-11 0:33 ` Josh Poimboeuf
2022-05-11 9:24 ` Petr Mladek
2022-05-11 16:33 ` Song Liu
2022-05-12 4:07 ` Josh Poimboeuf
2022-05-13 12:33 ` Petr Mladek
2022-05-13 13:34 ` Peter Zijlstra
2022-05-11 0:35 ` Rik van Riel
2022-05-11 0:37 ` Josh Poimboeuf
2022-05-11 0:46 ` Rik van Riel
2022-05-11 1:12 ` Josh Poimboeuf
2022-05-11 18:09 ` Rik van Riel
2022-05-12 3:59 ` Josh Poimboeuf
2022-05-09 15:52 ` [RFC] sched,livepatch: call stop_one_cpu in klp_check_and_switch_task Rik van Riel
2022-05-09 16:28 ` Song Liu
2022-05-09 18:00 ` Josh Poimboeuf
2022-05-09 19:10 ` Rik van Riel
2022-05-09 19:17 ` Josh Poimboeuf
2022-05-09 19:49 ` Rik van Riel
2022-05-09 20:09 ` Josh Poimboeuf
2022-05-10 0:32 ` Song Liu
2022-05-10 9:35 ` Peter Zijlstra
2022-05-10 1:48 ` Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=D298A3F1-43A5-4FD5-B198-906364BF4B79@fb.com \
--to=songliubraving@fb.com \
--cc=Kernel-team@fb.com \
--cc=joe.lawrence@redhat.com \
--cc=jpoimboe@kernel.org \
--cc=jpoimboe@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=live-patching@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=pmladek@suse.com \
--cc=riel@fb.com \
--cc=song@kernel.org \
--cc=vincent.guittot@linaro.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®