Re: next-20230110: arm64: defconfig+kselftest config boot failed - Unable to handle kernel paging request at virtual address fffffffffffffff8

Re: next-20230110: arm64: defconfig+kselftest config boot failed - Unable to handle kernel paging request at virtual address fffffffffffffff8

[Arnd Bergmann]

On Tue, Jan 10, 2023, at 17:14, Naresh Kamboju wrote:
> [ please ignore this email if this regression already reported ]
>
> Today's Linux next tag next-20230110 boot passes with defconfig but
> boot fails with
> defconfig + kselftest merge config on arm64 devices and qemu-arm64.
>
> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
>
> We are bisecting this problem and get back to you shortly.
>
> GOOD: next-20230109  (defconfig + kselftests configs)
> BAD: next-20230110 (defconfig + kselftests configs)
>
> kernel crash log [1]:
>
> [   15.302140] Unable to handle kernel paging request at virtual
> address fffffffffffffff8
> [   15.309906] Mem abort info:
> [   15.312659]   ESR = 0x0000000096000004
> [   15.316365]   EC = 0x25: DABT (current EL), IL = 32 bits
> [   15.321626]   SET = 0, FnV = 0
> [   15.324644]   EA = 0, S1PTW = 0
> [   15.327744]   FSC = 0x04: level 0 translation fault
> [   15.332619] Data abort info:
> [   15.335422]   ISV = 0, ISS = 0x00000004
> [   15.339226]   CM = 0, WnR = 0
> [   15.342154] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000001496c000
> [   15.348795] [fffffffffffffff8] pgd=0000000000000000, p4d=0000000000000000
> [   15.355524] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
> [   15.361729] Modules linked in: meson_gxl dwmac_generic
> snd_soc_meson_gx_sound_card snd_soc_meson_card_utils lima gpu_sched
> drm_shmem_helper meson_drm drm_dma_helper crct10dif_ce meson_ir
> rc_core meson_dw_hdmi dw_hdmi meson_canvas dwmac_meson8b
> stmmac_platform meson_rng stmmac rng_core cec meson_gxbb_wdt
> drm_display_helper snd_soc_meson_aiu snd_soc_meson_codec_glue pcs_xpcs
> snd_soc_meson_t9015 amlogic_gxl_crypto crypto_engine display_connector
> snd_soc_simple_amplifier drm_kms_helper drm nvmem_meson_efuse
> [   15.405976] CPU: 1 PID: 9 Comm: kworker/u8:0 Not tainted
> 6.2.0-rc3-next-20230110 #1
> [   15.413563] Hardware name: Libre Computer AML-S905X-CC (DT)
> [   15.419086] Workqueue: events_unbound deferred_probe_work_func
> [   15.424863] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [   15.431762] pc : of_drm_find_bridge+0x38/0x70 [drm]
> [   15.436594] lr : of_drm_find_bridge+0x20/0x70 [drm]

The line is 

drivers/gpu/drm/drm_bridge.c:1310:      if (bridge->of_node == np) {

The list_head here is a NULL pointer, so ->of_node points
to address negative 8, i.e. fffffffffffffff8

This is linked list corruption, which typically happens as
part of a use-after-free, and could be the result of a
failed registration causing an object to be freed after
it is added to the list.

Unfortunately, there are no patches to this file between
next-20230109 and next-20230110, so the bug probably is
not actually in this file.

> [   15.515426] Call trace:
> [   15.517863] Insufficient stack space to handle exception!
> [   15.517867] ESR: 0x0000000096000047 -- DABT (current EL)
> [   15.517871] FAR: 0xffff80000a047ff0
> [   15.517873] Task stack:     [0xffff80000a048000..0xffff80000a04c000]
> [   15.517877] IRQ stack:      [0xffff800008008000..0xffff80000800c000]
> [   15.517880] Overflow stack: [0xffff00007d9c1320..0xffff00007d9c2320]
> [   15.517884] CPU: 1 PID: 9 Comm: kworker/u8:0 Not tainted
> 6.2.0-rc3-next-20230110 #1
> [   15.517890] Hardware name: Libre Computer AML-S905X-CC (DT)
> [   15.517895] Workqueue: events_unbound deferred_probe_work_func
> [   15.517915] pstate: 800003c5 (Nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [   15.517923] pc : el1_abort+0x4/0x5c
> [   15.517932] lr : el1h_64_sync_handler+0x60/0xac
> [   15.517939] sp : ffff80000a048020

Not sure about the missing stack trace: I can see that the stack
pointer is on a task stack, which is reported as having overflown,
but I don't see why it's unable to print the stack while running
from the overflow stack.

A stack overflow is often caused by unbounded recursion, which
can happen when a device driver binds itself to a device that it
has just created. The log does look a bit suspicious here,
with multiple registrations for c883a000.hdmi-tx:

  986 08:02:56.487871  [   15.141218] meson-drm d0100000.vpu: Queued 2 outputs on vpu
  987 08:02:56.493572  [   15.141615] meson8b-dwmac c9410000.ethernet: Ring mode enabled
  988 08:02:56.504769  [   15.150744] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
  989 08:02:56.515743  [   15.154970] meson8b-dwmac c9410000.ethernet: Enable RX Mitigation via HW Watchdog Timer
  990 08:02:56.521531  [   15.159175] lima d00c0000.gpu: pp0 - mali450 version major 0 minor 0
  991 08:02:56.526718  [   15.161436] meson-drm d0100000.vpu: Failed to find HDMI transceiver bridge
  992 08:02:56.532417  [   15.168933] lima d00c0000.gpu: pp1 - mali450 version major 0 minor 0
  993 08:02:56.537747  [   15.206102] meson-drm d0100000.vpu: Queued 2 outputs on vpu
  994 08:02:56.543435  [   15.209608] lima d00c0000.gpu: pp2 - mali450 version major 0 minor 0
  995 08:02:56.554307  [   15.217027] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
  996 08:02:56.559929  [   15.221169] lima d00c0000.gpu: l2 cache 8K, 4-way, 64byte cache line, 128bit external bus
  997 08:02:56.565229  [   15.231561] meson-drm d0100000.vpu: Failed to find HDMI transceiver bridge
  998 08:02:56.576490  [   15.238133] lima d00c0000.gpu: l2 cache 64K, 4-way, 64byte cache line, 128bit external bus
  999 08:02:56.581759  [   15.253879] lima d00c0000.gpu: bus rate = 166666667
 1000 08:02:56.587248  [   15.257128] lima d00c0000.gpu: mod rate = 24000000
 1001 08:02:56.593344  [   15.261862] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
 1002 08:02:56.612809  [   15.286143] [drm] Initialized lima 1.1.0 20191231 for d00c0000.gpu on minor 1
 1003 08:02:56.618428  [   15.292259] meson-drm d0100000.vpu: Queued 2 outputs on vpu
 1004 08:02:56.623866  [   15.293715] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
 1005 08:02:56.634771  [   15.302140] Unable to handle kernel paging request at virtual address fffffffffffffff8

Again, I don't see any recent changes in that driver or the
corresponding dts file.

      Arnd

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: next-20230110: arm64: defconfig+kselftest config boot failed - Unable to handle kernel paging request at virtual address fffffffffffffff8

[Neil Armstrong]

Hi,

On 10/01/2023 17:41, Arnd Bergmann wrote:
> On Tue, Jan 10, 2023, at 17:14, Naresh Kamboju wrote:
>> [ please ignore this email if this regression already reported ]
>>
>> Today's Linux next tag next-20230110 boot passes with defconfig but
>> boot fails with
>> defconfig + kselftest merge config on arm64 devices and qemu-arm64.
>>
>> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
>>
>> We are bisecting this problem and get back to you shortly.
>>
>> GOOD: next-20230109  (defconfig + kselftests configs)
>> BAD: next-20230110 (defconfig + kselftests configs)
>>
>> kernel crash log [1]:
>>
>> [   15.302140] Unable to handle kernel paging request at virtual
>> address fffffffffffffff8
>> [   15.309906] Mem abort info:
>> [   15.312659]   ESR = 0x0000000096000004
>> [   15.316365]   EC = 0x25: DABT (current EL), IL = 32 bits
>> [   15.321626]   SET = 0, FnV = 0
>> [   15.324644]   EA = 0, S1PTW = 0
>> [   15.327744]   FSC = 0x04: level 0 translation fault
>> [   15.332619] Data abort info:
>> [   15.335422]   ISV = 0, ISS = 0x00000004
>> [   15.339226]   CM = 0, WnR = 0
>> [   15.342154] swapper pgtable: 4k pages, 48-bit VAs, pgdp=000000001496c000
>> [   15.348795] [fffffffffffffff8] pgd=0000000000000000, p4d=0000000000000000
>> [   15.355524] Internal error: Oops: 0000000096000004 [#1] PREEMPT SMP
>> [   15.361729] Modules linked in: meson_gxl dwmac_generic
>> snd_soc_meson_gx_sound_card snd_soc_meson_card_utils lima gpu_sched
>> drm_shmem_helper meson_drm drm_dma_helper crct10dif_ce meson_ir
>> rc_core meson_dw_hdmi dw_hdmi meson_canvas dwmac_meson8b
>> stmmac_platform meson_rng stmmac rng_core cec meson_gxbb_wdt
>> drm_display_helper snd_soc_meson_aiu snd_soc_meson_codec_glue pcs_xpcs
>> snd_soc_meson_t9015 amlogic_gxl_crypto crypto_engine display_connector
>> snd_soc_simple_amplifier drm_kms_helper drm nvmem_meson_efuse
>> [   15.405976] CPU: 1 PID: 9 Comm: kworker/u8:0 Not tainted
>> 6.2.0-rc3-next-20230110 #1
>> [   15.413563] Hardware name: Libre Computer AML-S905X-CC (DT)
>> [   15.419086] Workqueue: events_unbound deferred_probe_work_func
>> [   15.424863] pstate: 00000005 (nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [   15.431762] pc : of_drm_find_bridge+0x38/0x70 [drm]
>> [   15.436594] lr : of_drm_find_bridge+0x20/0x70 [drm]
> 
> The line is
> 
> drivers/gpu/drm/drm_bridge.c:1310:      if (bridge->of_node == np) {
> 
> The list_head here is a NULL pointer, so ->of_node points
> to address negative 8, i.e. fffffffffffffff8
> 
> This is linked list corruption, which typically happens as
> part of a use-after-free, and could be the result of a
> failed registration causing an object to be freed after
> it is added to the list.
> 
> Unfortunately, there are no patches to this file between
> next-20230109 and next-20230110, so the bug probably is
> not actually in this file.
> 
>> [   15.515426] Call trace:
>> [   15.517863] Insufficient stack space to handle exception!
>> [   15.517867] ESR: 0x0000000096000047 -- DABT (current EL)
>> [   15.517871] FAR: 0xffff80000a047ff0
>> [   15.517873] Task stack:     [0xffff80000a048000..0xffff80000a04c000]
>> [   15.517877] IRQ stack:      [0xffff800008008000..0xffff80000800c000]
>> [   15.517880] Overflow stack: [0xffff00007d9c1320..0xffff00007d9c2320]
>> [   15.517884] CPU: 1 PID: 9 Comm: kworker/u8:0 Not tainted
>> 6.2.0-rc3-next-20230110 #1
>> [   15.517890] Hardware name: Libre Computer AML-S905X-CC (DT)
>> [   15.517895] Workqueue: events_unbound deferred_probe_work_func
>> [   15.517915] pstate: 800003c5 (Nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
>> [   15.517923] pc : el1_abort+0x4/0x5c
>> [   15.517932] lr : el1h_64_sync_handler+0x60/0xac
>> [   15.517939] sp : ffff80000a048020
> 
> Not sure about the missing stack trace: I can see that the stack
> pointer is on a task stack, which is reported as having overflown,
> but I don't see why it's unable to print the stack while running
> from the overflow stack.
> 
> A stack overflow is often caused by unbounded recursion, which
> can happen when a device driver binds itself to a device that it
> has just created. The log does look a bit suspicious here,
> with multiple registrations for c883a000.hdmi-tx:
> 
>    986 08:02:56.487871  [   15.141218] meson-drm d0100000.vpu: Queued 2 outputs on vpu
>    987 08:02:56.493572  [   15.141615] meson8b-dwmac c9410000.ethernet: Ring mode enabled
>    988 08:02:56.504769  [   15.150744] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
>    989 08:02:56.515743  [   15.154970] meson8b-dwmac c9410000.ethernet: Enable RX Mitigation via HW Watchdog Timer
>    990 08:02:56.521531  [   15.159175] lima d00c0000.gpu: pp0 - mali450 version major 0 minor 0
>    991 08:02:56.526718  [   15.161436] meson-drm d0100000.vpu: Failed to find HDMI transceiver bridge
>    992 08:02:56.532417  [   15.168933] lima d00c0000.gpu: pp1 - mali450 version major 0 minor 0
>    993 08:02:56.537747  [   15.206102] meson-drm d0100000.vpu: Queued 2 outputs on vpu
>    994 08:02:56.543435  [   15.209608] lima d00c0000.gpu: pp2 - mali450 version major 0 minor 0
>    995 08:02:56.554307  [   15.217027] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
>    996 08:02:56.559929  [   15.221169] lima d00c0000.gpu: l2 cache 8K, 4-way, 64byte cache line, 128bit external bus
>    997 08:02:56.565229  [   15.231561] meson-drm d0100000.vpu: Failed to find HDMI transceiver bridge
>    998 08:02:56.576490  [   15.238133] lima d00c0000.gpu: l2 cache 64K, 4-way, 64byte cache line, 128bit external bus
>    999 08:02:56.581759  [   15.253879] lima d00c0000.gpu: bus rate = 166666667
>   1000 08:02:56.587248  [   15.257128] lima d00c0000.gpu: mod rate = 24000000
>   1001 08:02:56.593344  [   15.261862] lima d00c0000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
>   1002 08:02:56.612809  [   15.286143] [drm] Initialized lima 1.1.0 20191231 for d00c0000.gpu on minor 1
>   1003 08:02:56.618428  [   15.292259] meson-drm d0100000.vpu: Queued 2 outputs on vpu
>   1004 08:02:56.623866  [   15.293715] meson-drm d0100000.vpu: bound c883a000.hdmi-tx (ops meson_dw_hdmi_ops [meson_dw_hdmi])
>   1005 08:02:56.634771  [   15.302140] Unable to handle kernel paging request at virtual address fffffffffffffff8
> 
> Again, I don't see any recent changes in that driver or the
> corresponding dts file.

I merged a fix that could be related: https://lore.kernel.org/all/20230109220033.31202-1-m.szyprowski@samsung.com/

This could make the driver to return from probe while not totally probed, and explain such error.

Neil

> 
>        Arnd


_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: next-20230110: arm64: defconfig+kselftest config boot failed - Unable to handle kernel paging request at virtual address fffffffffffffff8

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 1048 bytes --]

On Wed, Jan 11, 2023 at 11:34:41AM +0100, Neil Armstrong wrote:

> I merged a fix that could be related: https://lore.kernel.org/all/20230109220033.31202-1-m.szyprowski@samsung.com/

> This could make the driver to return from probe while not totally probed, and explain such error.

We're seeing issues in all configs on meson-gxl-s905x-libretech-cc
today, not just with the kselftest fragment.  The initial failuire seems
to be:

[   17.337253] WARNING: CPU: 3 PID: 123 at drivers/gpu/drm/drm_bridge.c:1257 drm_bridge_hpd_enable+0x8c/0x94 [drm]

full log at:

   https://storage.kernelci.org/next/master/next-20230111/arm64/defconfig/gcc-10/lab-broonie/baseline-meson-gxl-s905x-libretech-cc.txt

and links to other logs at:

  https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20230111/plan/baseline/

Today's -next does have that fix in it so it's not fixing whatever the
original issue was, I suspect it might even be exposing other issues.
We are however still seeing the stack filling up, even with a GCC 10
defconfig build.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 167 bytes --]

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: next-20230110: arm64: defconfig+kselftest config boot failed - Unable to handle kernel paging request at virtual address fffffffffffffff8

[Mark Brown]

[-- Attachment #1.1: Type: text/plain, Size: 3931 bytes --]

On Wed, Jan 11, 2023 at 12:29:04PM +0000, Mark Brown wrote:

> We're seeing issues in all configs on meson-gxl-s905x-libretech-cc
> today, not just with the kselftest fragment.  The initial failuire seems
> to be:

> [   17.337253] WARNING: CPU: 3 PID: 123 at drivers/gpu/drm/drm_bridge.c:1257 drm_bridge_hpd_enable+0x8c/0x94 [drm]

> full log at:

>    https://storage.kernelci.org/next/master/next-20230111/arm64/defconfig/gcc-10/lab-broonie/baseline-meson-gxl-s905x-libretech-cc.txt

> and links to other logs at:

>   https://linux.kernelci.org/test/job/next/branch/master/kernel/next-20230111/plan/baseline/

> Today's -next does have that fix in it so it's not fixing whatever the
> original issue was, I suspect it might even be exposing other issues.
> We are however still seeing the stack filling up, even with a GCC 10
> defconfig build.

A bisect landed on 0e4dcffd331fa7d ("drm/panel: raspberrypi-touchscreen:
Convert to i2c's .probe_new()") which is obviously not credible.  I
suspect that what's happening here is that the fix you applied is making
an issue somewhere else visible in defconfig and is as a result
confusing the bisect.  Ard mentioned an issue with non-EFI biits
introduced by EFI changes here:

https://lore.kernel.org/linux-arm-kernel/CAMj1kXGFa=Zriyp_mS7bbQr0wiwikt0ObjOKUSNGpJtFvLmnkg@mail.gmail.com/

which seems like a plausible culprit,

bisect log:

git bisect start
# bad: [c9e9cdd8bdcc3e1ea330d49ea587ec71884dd0f5] Add linux-next specific files for 20230111
git bisect bad c9e9cdd8bdcc3e1ea330d49ea587ec71884dd0f5
# good: [7dd4b804e08041ff56c88bdd8da742d14b17ed25] Merge tag 'nfsd-6.2-3' of git://git.kernel.org/pub/scm/linux/kernel/git/cel/linux
git bisect good 7dd4b804e08041ff56c88bdd8da742d14b17ed25
# good: [ecf8827ab7dd5731813f90146d9936165b170f32] Merge branch 'drm-next' of git://git.freedesktop.org/git/drm/drm.git
git bisect good ecf8827ab7dd5731813f90146d9936165b170f32
# bad: [64208e4940ede76709f1ff5b01d1b78efc2951cf] Merge branch 'rcu/next' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git
git bisect bad 64208e4940ede76709f1ff5b01d1b78efc2951cf
# bad: [1077dd31ba60b39a231560beec24b97eadf8bd8f] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git
git bisect bad 1077dd31ba60b39a231560beec24b97eadf8bd8f
# bad: [1577a2c2aad943fbc6a5e959ae83c4ef8bc3d4de] Merge branch 'drm-next' of https://gitlab.freedesktop.org/agd5f/linux
git bisect bad 1577a2c2aad943fbc6a5e959ae83c4ef8bc3d4de
# good: [ec787deb2ddffc6cd6afe0e2fbbbd490ddc383ed] drm/amd: Use `amdgpu_ucode_*` helpers for GFX9
git bisect good ec787deb2ddffc6cd6afe0e2fbbbd490ddc383ed
# bad: [0e4dcffd331fa7d2a6ae628b51a7f418dfa90367] drm/panel: raspberrypi-touchscreen: Convert to i2c's .probe_new()
git bisect bad 0e4dcffd331fa7d2a6ae628b51a7f418dfa90367
# good: [c702545e19ebb6113d607f2a30ba2ee6cf881a3a] drm/gud: use new debugfs device-centered functions
git bisect good c702545e19ebb6113d607f2a30ba2ee6cf881a3a
# good: [977374cf481d3bea916b2775e6ecc682b9689550] drm/vc4: plane: Add 3:3:2 and 4:4:4:4 RGB/RGBX/RGBA formats
git bisect good 977374cf481d3bea916b2775e6ecc682b9689550
# good: [67d0a30128c9f644595dfe67ac0fb941a716a6c9] drm/meson: dw-hdmi: Fix devm_regulator_*get_enable*() conversion
git bisect good 67d0a30128c9f644595dfe67ac0fb941a716a6c9
# good: [29ef7605e2fd44038a70df0f46b7821464081b22] drm/i2c/sil164: Convert to i2c's .probe_new()
git bisect good 29ef7605e2fd44038a70df0f46b7821464081b22
# good: [307259952625798fbea89b04aebbc5106ff18c68] drm/i2c/tda998x: Convert to i2c's .probe_new()
git bisect good 307259952625798fbea89b04aebbc5106ff18c68
# good: [446757576a646eba6fae085396bdfbd74245ff28] drm/panel: olimex-lcd-olinuxino: Convert to i2c's .probe_new()
git bisect good 446757576a646eba6fae085396bdfbd74245ff28
# first bad commit: [0e4dcffd331fa7d2a6ae628b51a7f418dfa90367] drm/panel: raspberrypi-touchscreen: Convert to i2c's .probe_new()

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[-- Attachment #2: Type: text/plain, Size: 167 bytes --]

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic