From: "Arnd Bergmann" <arnd@arndb.de>
To: "Naresh Kamboju" <naresh.kamboju@linaro.org>,
"Anders Roxell" <anders.roxell@linaro.org>,
"Daniel Díaz" <daniel.diaz@linaro.org>,
"Benjamin Copeland" <ben.copeland@linaro.org>
Cc: "Peter Zijlstra" <peterz@infradead.org>,
linux-kernel@vger.kernel.org, x86@kernel.org
Subject: Re: qemu-x86_64 booting with 8.0.0 stil see int3: when running LTP tracing testing.
Date: Wed, 21 Jun 2023 17:31:15 +0200 [thread overview]
Message-ID: <2d7595b1-b655-4425-85d3-423801bce644@app.fastmail.com> (raw)
In-Reply-To: <CA+G9fYsETJQm0Ue7hGsb+nbsiMikwycOV3V0DPr6WC2r61KRBQ@mail.gmail.com>
On Wed, Jun 21, 2023, at 16:16, Naresh Kamboju wrote:
> Hi Team,
>
> FYI,
> qemu-x86_64 booting with 8.0.0 still see int3: when running LTP tracing testing.
>
> docker.io/linaro/tuxrun-qemu:v8.0.0 qemu-system-x86_64
Thanks for the report. I've added the x86 and kernel lists as well
as Peter Zijlstra to Cc.
> Running tests.......
> ftrace_regression01 1 TPASS: Finished running the test
> <4>[ 49.874424] int3: 0000 [#1] PREEMPT SMP PTI
> <4>[ 49.874634] int3: 0000 [#2] PREEMPT SMP PTI
> <4>[ 49.874656] int3: 0000 [#3] PREEMPT SMP PTI
> <4>[ 49.874669] int3: 0000 [#4] PREEMPT SMP PTI
> <4>[ 49.874680] int3: 0000 [#5] PREEMPT SMP PTI
> <4>[ 49.874691] int3: 0000 [#6] PREEMPT SMP PTI
> <4>[ 49.874703] int3: 0000 [#7] PREEMPT SMP PTI
> <4>[ 49.874714] int3: 0000 [#8] PREEMPT SMP PTI
> <4>[ 49.874725] int3: 0000 [#9] PREEMPT SMP PTI
At first looked a bit like it's hitting on multiple CPUs, but
I also see that the instance only has two CPUs, so apparently
this instance is hitting the same bug recursively from
the exception handler.
> <4>[ 49.874736] int3: 0000 [#10] PREEMPT SMP PTI
> <4>[ 49.874750] int3: 0000 [#11] PREEMPT SMP PTI
> <4>[ 49.874761] int3: 0000 [#12] PREEMPT SMP PTI
> <4>[ 49.874772] int3: 0000 [#13] PREEMPT SMP PTI
> <4>[ 49.874783] int3: 0000 [#14] PREEMPT SMP PTI
> <4>[ 49.874828] int3: 0000 [#15] PREEMPT SMP PTI
> <4>[ 49.874894] int3: 0000 [#16] PREEMPT SMP PTI
> <4>[ 49.874907] int3: 0000 [#17] PREEMPT SMP PTI
> <4>[ 49.874918] int3: 0000 [#18] PREEMPT SMP PTI
> <4>[ 49.874930] int3: 0000 [#19] PREEMPT SMP PTI
> <4>[ 49.874941] int3: 0000 [#20] PREEMPT SMP PTI
> <4>[ 49.874953] int3: 0000 [#21] PREEMPT SMP PTI
> <4>[ 49.874964] int3: 0000 [#22] PREEMPT SMP PTI
> <4>[ 49.874976] int3: 0000 [#23] PREEMPT SMP PTI
> <4>[ 49.874987] int3: 0000 [#24] PREEMPT SMP PTI
> <4>[ 49.874999] int3: 0000 [#25] PREEMPT SMP PTI
> <4>[ 49.875010] int3: 0000 [#26] PREEMPT SMP PTI
> <4>[ 49.875021] int3: 0000 [#27] PREEMPT SMP PTI
> <4>[ 49.875032] int3: 0000 [#28] PREEMPT SMP PTI
> <4>[ 49.875043] int3: 0000 [#29] PREEMPT SMP PTI
> <4>[ 49.875054] int3: 0000 [#30] PREEMPT SMP PTI
> <4>[ 49.875282] ------------[ cut here ]------------
> <4>[ 49.875687] ------------[ cut here ]------------
> <2>[ 49.875904] kernel BUG at kernel/entry/common.c:454!
> <4>[ 49.876652] invalid opcode: 0000 [#31] PREEMPT SMP PTI
> <4>[ 49.876845] CPU: 1 PID: 317 Comm: ftrace_regressi Not tainted
> 6.4.0-rc7-next-20230621 #1
> <4>[ 49.876994] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014
> <4>[ 49.877119] RIP: 0010:irqentry_nmi_enter+0x68/0x70
> <4>[ 49.877756] Code: 48 8b 0d 7b 3a 6a 73 c7 81 88 08 00 00 00 00
> 00 00 85 c0 0f 95 c3 e8 27 02 00 00 e8 22 93 fe fe 89 d8 5b 5d c3 cc
> cc cc cc cc <0f> 0b 66 0f 1f 44 00 00 90 90 90 90 90 90 90 90 90 90 90
> 90 90 90
> <4>[ 49.877929] RSP: 0000:fffffe3e79ab6ee8 EFLAGS: 00000046
> <4>[ 49.878425] RAX: 0000000000000000 RBX: fffffe3e79ab6f58 RCX:
> 00000000fe00ffdb
> <4>[ 49.878444] RDX: 00000000ffffa445 RSI: 0000000000000000 RDI:
> fffffe3e79ab6f58
> <4>[ 49.878460] RBP: fffffe3e79ab6ef0 R08: 0000000000000000 R09:
> 0000000000000000
> <4>[ 49.878475] R10: 0000000000000000 R11: 0000000000000000 R12:
> ffffa44502ab5a00
> <4>[ 49.878490] R13: 0000000000000000 R14: 0000000000000000 R15:
> ffffab17000ccfd8
> <4>[ 49.878540] FS: 00007f91069b4740(0000)
> GS:ffffa4457bd00000(0000) knlGS:0000000000000000
> <4>[ 49.878579] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> <4>[ 49.878596] CR2: ffffab17000ccfd8 CR3: 0000000102be8000 CR4:
> 00000000000006e0
> <4>[ 49.878695] Call Trace:
> <4>[ 49.878892] <#DF>
> <4>[ 49.879037] ? __die_body+0x6c/0xc0
> <4>[ 49.879123] ? die+0xae/0xe0
> <4>[ 49.879151] ? do_trap+0x8d/0x160
> <4>[ 49.879178] ? irqentry_nmi_enter+0x68/0x70
> <4>[ 49.879199] ? handle_invalid_op+0x7f/0xd0
> <4>[ 49.879213] ? irqentry_nmi_enter+0x68/0x70
> <4>[ 49.879237] ? exc_invalid_op+0x36/0x50
> <4>[ 49.879259] ? asm_exc_invalid_op+0x1f/0x30
> <4>[ 49.879316] ? irqentry_nmi_enter+0x68/0x70
> <4>[ 49.879344] exc_double_fault+0x107/0x1b0
> <4>[ 49.879416] asm_exc_double_fault+0x23/0x30
> <4>[ 49.879479] RIP: 0010:0xffffffffc00ac0ac
I don't know much about x86 exception handling, but my guess is
that this is where the stack overflows, so this backtrace
is not all that useful. Looking at the full log from your link,
I see that recursion through asm_exc_int3:
<4>[ 49.886694] ? exc_int3+0x62/0x80
<4>[ 49.886714] ? asm_exc_int3+0x3e/0x50
<4>[ 49.886759] ? preempt_count_sub+0x5/0x80
<4>[ 49.886783] ? preempt_count_sub+0x5/0x80
<4>[ 49.886805] ? irq_work_queue+0x40/0x80
<4>[ 49.886826] ? defer_console_output+0x49/0x80
<4>[ 49.886840] ? vprintk+0x42/0x60
<4>[ 49.886857] ? _printk+0x5d/0x80
<4>[ 49.886891] ? die+0x9c/0xe0
<4>[ 49.886922] ? exc_int3+0x62/0x80
<4>[ 49.886942] ? asm_exc_int3+0x3e/0x50
<4>[ 49.886966] ? __pfx_tick_sched_timer+0x10/0x10
<4>[ 49.886980] ? __pfx_read_tsc+0x10/0x10
<4>[ 49.887010] ? preempt_count_sub+0x5/0x80
<4>[ 49.887035] ? preempt_count_sub+0x5/0x80
<4>[ 49.887057] ? __hrtimer_run_queues+0xee/0x330
<4>[ 49.887070] ? _raw_spin_unlock_irqrestore+0x28/0x50
<4>[ 49.887085] ? __hrtimer_run_queues+0xee/0x330
<4>[ 49.887136] ? hrtimer_interrupt+0xf6/0x390
<4>[ 49.887167] ? __sysvec_apic_timer_interrupt+0x64/0x1a0
<4>[ 49.887189] ? sysvec_apic_timer_interrupt+0x7a/0x90
<4>[ 49.887206] </IRQ>
<4>[ 49.887219] <TASK>
<4>[ 49.887231] ? asm_sysvec_apic_timer_interrupt+0x1f/0x30
<4>[ 49.887252] ? sched_rt_period_timer+0x4/0x390
<4>[ 49.887272] ? __pfx_do_sync_core+0x10/0x10
<4>[ 49.887305] ? insn_get_displacement+0x9/0x160
<4>[ 49.887329] ? insn_get_displacement+0x9/0x160
<4>[ 49.887344] ? insn_get_immediate+0xd2/0x270
<4>[ 49.887365] ? insn_decode+0x113/0x150
<4>[ 49.887384] ? text_poke_loc_init+0xea/0x220
<4>[ 49.887429] ? sched_rt_period_timer+0x4/0x390
<4>[ 49.887451] ? text_poke_queue+0x89/0xa0
<4>[ 49.887477] ? ftrace_replace_code+0x149/0x1f0
<4>[ 49.887508] ? ftrace_modify_all_code+0x71/0x140
<4>[ 49.887533] ? arch_ftrace_update_code+0xd/0x20
<4>[ 49.887550] ? ftrace_shutdown+0xf5/0x220
<4>[ 49.887577] ? unregister_ftrace_function+0x2e/0x150
<4>[ 49.887608] ? stack_trace_sysctl+0x82/0xb0
<4>[ 49.887636] ? proc_sys_call_handler+0x18b/0x280
<4>[ 49.887675] ? proc_sys_write+0x17/0x20
<4>[ 49.887692] ? vfs_write+0x324/0x3f0
<4>[ 49.887742] ? ksys_write+0x75/0xe0
<4>[ 49.887771] ? __x64_sys_write+0x1f/0x30
<4>[ 49.887787] ? do_syscall_64+0x48/0xa0
<4>[ 49.887802] ? sysvec_apic_timer_interrupt+0x4d/0x90
<4>[ 49.887822] ? entry_SYSCALL_64_after_hwframe+0x6e/0xd8
<4>[ 49.887878] </TASK>
This looks like a timer interrupt happens inside of
text_poke_loc_init(), i.e. while the kernel is modifying
itself, and presumably adding (or removing) an int3
instruction that is later hit inside of the timer function.
While our previous theory was that this was most likely
a qemu bug in dealing with self-modifying code, this might
actually hint at a problem in the kernel after all.
I think Peter understands this function best, he probably
sees more here than I do.
> Link:
> https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20230621/testrun/17699662/suite/log-parser-test/tests/
>
> https://qa-reports.linaro.org/lkft/linux-next-master/build/next-20230621/testrun/17699662/suite/log-parser-test/test/check-kernel-panic/log
>
> - Naresh
Arnd
next parent reply other threads:[~2023-06-21 15:32 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CA+G9fYsETJQm0Ue7hGsb+nbsiMikwycOV3V0DPr6WC2r61KRBQ@mail.gmail.com>
2023-06-21 15:31 ` Arnd Bergmann [this message]
2023-06-21 16:06 ` Peter Zijlstra
2023-07-04 7:46 ` Richard W.M. Jones
2023-07-04 13:21 ` Richard W.M. Jones
2023-07-05 16:28 ` Richard W.M. Jones
[not found] ` <CAFXwXrmbpuFNf5=nQxiTteo8fpCdAbK4pEAN176Cq0yvwZcfFw@mail.gmail.com>
2023-07-05 16:35 ` Richard W.M. Jones
[not found] ` <CAFXwXrk1FEZPUO-zqNVJZ6YCHKUkgNehwmyDYuOr5fx8ff0OCA@mail.gmail.com>
2023-07-05 16:40 ` Richard W.M. Jones
2023-07-06 6:13 ` Richard Henderson
2023-07-05 16:37 ` Richard W.M. Jones
2023-07-05 21:50 ` Richard W.M. Jones
2023-07-06 6:30 ` Richard Henderson
2023-07-06 10:28 ` Richard W.M. Jones
2023-08-08 5:27 ` John Stultz
2023-08-08 7:28 ` Richard W.M. Jones
2025-10-25 19:48 ` Gregory Price
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2d7595b1-b655-4425-85d3-423801bce644@app.fastmail.com \
--to=arnd@arndb.de \
--cc=anders.roxell@linaro.org \
--cc=ben.copeland@linaro.org \
--cc=daniel.diaz@linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=naresh.kamboju@linaro.org \
--cc=peterz@infradead.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®