Re: [PATCH] xen/scsiback: unbind the IRQ before draining in-flight requests

[Juergen Gross <jgross@suse.com>]

[-- Attachment #1.1: Type: text/plain, Size: 8014 bytes --]

On 09.11.26 13:26, Yehyeong Lee wrote:
> scsiback_disconnect() drains the in-flight count before stopping the only
> thing that can raise it.  scsiback_cmd_exec() (xen-scsiback.c:426) runs in
> the backend's threaded IRQ handler, so a request can be admitted just after
> wait_event() has seen zero, and its response is written later from target
> core context by scsiback_send_response() (:347) - after the ring has been
> unmapped, and on the scsiback_remove() path (:1268-1280) after kfree(info).
> 
> xen-blkback and xen-netback stop their request processing first:
> kthread_stop() before reading ring->inflight
> (xen-blkback/xenbus.c:290-304), and the kthreads and napi before
> unbinding and unmapping (xen-netback/interface.c:673-700).
> 
> Unbind and clear info->irq before the drain: nothing is admitted once the
> handler is gone, and the ring stays mapped until the drain finishes.  The
> notify at the end of scsiback_send_response() is then a no-op, because
> notify_remote_via_irq() resolves irq 0 through evtchn_from_irq() to port 0,
> which VALID_EVTCHN() rejects (events_base.c:539-545, :411-421).
> 
> Reproduced in a nested-Xen lab with KASAN.  Reaching the window needed two
> test-only usleep_range() delays, one after wait_event() returns and one at
> the top of scsiback_do_resp_with_sense() once the teardown has started.
> With both off neither arm produced anything in 4 rounds each; with both on
> the unpatched driver crashed in 6 of 6 valid cells, the patched in 0 of 7:
> 
> [  168.546709] BUG: KASAN: vmalloc-out-of-bounds in scsiback_send_response+0x516/0x5d0
> [  168.551012] Write of size 4 at addr ffffc900000230a4 by task kworker/0:1/11
> [  168.554757]
> [  168.555908] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Not tainted 7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full)
> [  168.555914] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [  168.555918] Workqueue: target_completion target_complete_ok_work
> [  168.555939] Call Trace:
> [  168.555946]  <TASK>
> [  168.555949]  dump_stack_lvl+0x64/0x80
> [  168.555958]  print_report+0xd0/0x630
> [  168.555966]  ? __pfx__raw_spin_lock_irqsave+0x10/0x10
> [  168.555978]  ? scsiback_send_response+0x516/0x5d0
> [  168.555983]  kasan_report+0xce/0x100
> [  168.555988]  ? scsiback_send_response+0x516/0x5d0
> [  168.555992]  scsiback_send_response+0x516/0x5d0
> [  168.555997]  ? __pfx_scsiback_send_response+0x10/0x10
> [  168.556001]  scsiback_do_resp_with_sense+0x87/0x230
> [  168.556007]  scsiback_cmd_done+0xc0/0x290
> [  168.556011]  ? update_cfs_rq_load_avg+0x5a/0x560
> [  168.556023]  scsiback_queue_status+0xd3/0x100
> [  168.556028]  target_complete_ok_work+0x467/0xc30
> [  168.556032]  ? __pfx_target_complete_ok_work+0x10/0x10
> [  168.556036]  ? put_prev_task_fair+0x44d/0xa30
> [  168.556039]  ? finish_task_switch.isra.0+0x1d9/0xa70
> [  168.556047]  process_one_work+0x697/0x1060
> [  168.556052]  ? __pfx___schedule+0x10/0x10
> [  168.556058]  ? __pfx_process_one_work+0x10/0x10
> [  168.556061]  ? _raw_spin_lock_irq+0x86/0xe0
> [  168.556065]  ? __pfx_target_complete_ok_work+0x10/0x10
> [  168.556069]  ? assign_work+0x11d/0x370
> [  168.556074]  worker_thread+0x45b/0xd10
> [  168.556078]  ? __pfx_worker_thread+0x10/0x10
> [  168.556081]  ? __pfx_worker_thread+0x10/0x10
> [  168.556083]  kthread+0x2c8/0x3b0
> [  168.556090]  ? recalc_sigpending+0x15c/0x1e0
> [  168.556094]  ? __pfx_kthread+0x10/0x10
> [  168.556098]  ret_from_fork+0x373/0x5b0
> [  168.556104]  ? __pfx_ret_from_fork+0x10/0x10
> [  168.556107]  ? __switch_to+0x3c4/0xb40
> [  168.556115]  ? __pfx_kthread+0x10/0x10
> [  168.556120]  ret_from_fork_asm+0x1a/0x30
> [  168.556127]  </TASK>
> [  168.556129]
> [  168.639704] The buggy address belongs to a vmalloc virtual mapping
> [  168.642324] Memory state around the buggy address:
> [  168.644439]  ffffc90000022f80: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
> [  168.647698]  ffffc90000023000: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
> [  168.650951] >ffffc90000023080: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
> [  168.669199]                                ^
> [  168.671056]  ffffc90000023100: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
> [  168.674050]  ffffc90000023180: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
> [  168.676974] ==================================================================
> 
> [  168.682117] BUG: unable to handle page fault for address: ffffc900000230a4
> [  168.684784] #PF: supervisor write access in kernel mode
> [  168.686797] #PF: error_code(0x0002) - not-present page
> [  168.688797] PGD 100000067 P4D 100000067 PUD 10019c067 PMD 1001a1067 PTE 0
> [  168.691319] Oops: Oops: 0002 [#1] SMP KASAN NOPTI
> [  168.693246] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Tainted: G    B                7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full)
> [  168.699142] Tainted: [B]=BAD_PAGE
> [  168.700896] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014
> [  168.705056] Workqueue: target_completion target_complete_ok_work
> [  168.707701] RIP: 0010:scsiback_send_response+0x1b6/0x5d0
> [  168.710147] Code: 48 89 f8 83 e0 07 83 c0 03 40 38 f0 7c 09 40 84 f6 0f 85 60 03 00 00 48 89 d8 48 c1 e0 06 48 29 d8 48 c1 e0 02 48 8d 7c 01 40 <89> 94 01 a4 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1
> [  168.718883] RSP: 0018:ffffc900000bfaa0 EFLAGS: 00010046
> [  168.721447] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffc90000023000
> [  168.724785] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc90000023040
> [  168.727913] RBP: ffff88810a4a7238 R08: 0000000000000246 R09: fffffbfff0de2638
> [  168.730938] R10: ffffffff86f131c7 R11: ffff888100077320 R12: 1ffff92000017f59
> [  168.733902] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88810a6de200
> [  168.736874] FS:  0000000000000000(0000) GS:ffff8881b034d000(0000) knlGS:0000000000000000
> [  168.740171] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  168.742490] CR2: ffffc900000230a4 CR3: 0000000103271005 CR4: 0000000000770ef0
> [  168.745350] PKRU: 55555554
> [  168.746577] Call Trace:
> [  168.747772]  <TASK>
> [  168.748872]  ? __pfx_scsiback_send_response+0x10/0x10
> [  168.751022]  scsiback_do_resp_with_sense+0x87/0x230
> [  168.753109]  scsiback_cmd_done+0xc0/0x290
> [  168.754847]  ? update_cfs_rq_load_avg+0x5a/0x560
> [  168.756782]  scsiback_queue_status+0xd3/0x100
> [  168.758630]  target_complete_ok_work+0x467/0xc30
> [  168.760556]  ? __pfx_target_complete_ok_work+0x10/0x10
> [  168.762644]  ? put_prev_task_fair+0x44d/0xa30
> [  168.764512]  ? finish_task_switch.isra.0+0x1d9/0xa70
> [  168.766584]  process_one_work+0x697/0x1060
> [  168.768282]  ? __pfx___schedule+0x10/0x10
> [  168.769973]  ? __pfx_process_one_work+0x10/0x10
> [  168.771880]  ? _raw_spin_lock_irq+0x86/0xe0
> [  168.773599]  ? __pfx_target_complete_ok_work+0x10/0x10
> [  168.775721]  ? assign_work+0x11d/0x370
> [  168.777324]  worker_thread+0x45b/0xd10
> [  168.778899]  ? __pfx_worker_thread+0x10/0x10
> [  168.780648]  ? __pfx_worker_thread+0x10/0x10
> [  168.782415]  kthread+0x2c8/0x3b0
> [  168.783877]  ? recalc_sigpending+0x15c/0x1e0
> [  168.785697]  ? __pfx_kthread+0x10/0x10
> [  168.787218]  ret_from_fork+0x373/0x5b0
> [  168.788752]  ? __pfx_ret_from_fork+0x10/0x10
> [  168.790500]  ? __switch_to+0x3c4/0xb40
> [  168.792029]  ? __pfx_kthread+0x10/0x10
> [  168.793555]  ret_from_fork_asm+0x1a/0x30
> [  168.795180]  </TASK>
> [  168.796205] Modules linked in:
> [  168.797596] CR2: ffffc900000230a4
> [  168.799084] ---[ end trace 0000000000000000 ]---
> 
> Fixes: d9d660f6e562 ("xen-scsiback: Add Xen PV SCSI backend driver")
> Cc: stable@vger.kernel.org
> Signed-off-by: Yehyeong Lee <yhlee@isslab.korea.ac.kr>
Reviewed-by: Juergen Gross <jgross@suse.com>


Juergen

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]