Re: Preemptable Ticket Spinlock

[Andrew Theurer <habanero@linux.vnet.ibm.com>]

On Sat, 2013-04-20 at 18:12 -0400, Jiannan Ouyang wrote:
> Hello Everyone,
> 
> 
> I recently came up with a spinlock algorithm that can adapt to
> preemption, which you may be interested in. The intuition is to
> downgrade a fair lock to an unfair lock automatically upon preemption,
> and preserve the fairness otherwise. It is a guest side optimization,
> and can be used as a complementary technique to host side
> optimizations like co-scheduling and Pause-Loop Exiting.
> 
> 
> In my experiments, it improves VM performance by 5:32X on average,
> when running on a non paravirtual VMM, and by 7:91X when running on a
> VMM that supports a paravirtual locking interface (using a pv
> preemptable ticket spinlock), when executing a set of microbenchmarks
> as well as a realistic e-commerce benchmark.
> 
> 
> A detailed algorithm description can be found in my VEE 2013 paper,
> Preemptable Ticket Spinlocks: Improving Consolidated Performance in
> the Cloud
> Jiannan Ouyang, John R. Lange
> ouyang,jacklange@cs.pitt.edu
> University of Pittsburgh
> http://people.cs.pitt.edu/~ouyang/files/publication/preemptable_lock-ouyang-vee13.pdf
> 
> 
> The patch is based on stock Linux kernel 3.5.0, and tested on kernel
> 3.4.41 as well.
> http://www.cs.pitt.edu/~ouyang/files/preemptable_lock.tar.gz

Very nice paper.  I wanted to see how this would work on larger VMs on
the dbench workload.  Unfortunately, when I tried your patch on 3.9-rc8
+, I get a lot of CPU soft lockup messages from the VMs, to the point
where the test cannot complete in a reasonable amount of time:


> [ 2144.672812] BUG: soft lockup - CPU#16 stuck for 23s! [dbench:8618]
> [ 2144.672888] Modules linked in: bridge stp llc target_core_mod configfs autofs4 sunrpc af_packet ipv6 binfmt_misc dm_mirror dm_region_hash dm_log dm_mod uinput rtc_cmos button crc32c_intel microcode pcspkr virtio_net i2c_piix4 i2c_core intel_agp intel_gtt ext4 mbcache jbd2 crc16 virtio_blk floppy aesni_intel ablk_helper cryptd lrw aes_x86_64 xts gf128mul virtio_pci virtio_ring virtio uhci_hcd usbcore usb_common pata_acpi ata_generic piix ide_core ata_piix libata scsi_mod [last unloaded: mperf]
> [ 2144.672892] CPU 16
> [ 2144.672892] Pid: 8618, comm: dbench Not tainted 3.9.0-rc8-soft-ticket-0.27-default+ #3 Bochs Bochs
> [ 2144.672898] RIP: 0010:[<ffffffff814a0c93>]  [<ffffffff814a0c93>] _raw_spin_unlock_irqrestore+0x13/0x20
> [ 2144.672899] RSP: 0018:ffff8807c0203d68  EFLAGS: 00000202
> [ 2144.672901] RAX: 00000000ffffffff RBX: ffff8807c0203ce8 RCX: ffff8807bfe13244
> [ 2144.672902] RDX: 0000000000000001 RSI: 0000000000000202 RDI: 0000000000000202
> [ 2144.672903] RBP: ffff8807c0203d68 R08: 0000000000001774 R09: 0000000000001777
> [ 2144.672904] R10: 0000000000000001 R11: 0000000000ef2400 R12: ffff8807c0203cd8
> [ 2144.672906] R13: ffffffff814aa09d R14: ffff8807c0203d68 R15: 0000000000000000
> [ 2144.672907] FS:  00007f42edc22700(0000) GS:ffff8807c0200000(0000) knlGS:0000000000000000
> [ 2144.672908] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 2144.672908] CR2: 00007f021f6bdc30 CR3: 000000079c63b000 CR4: 00000000000006e0
> [ 2144.673131] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 2144.673182] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 2144.673184] Process dbench (pid: 8618, threadinfo ffff8807a0394000, task ffff880790c22340)
> [ 2144.673185] Stack:
> [ 2144.673189]  ffff8807c0203e68 ffffffff81086713 0000000000000092 0000000000013240
> [ 2144.673295]  0000000000013240 0000001000000092 0000000000000025 ffff8807c0203ea4
> [ 2144.673300]  ffff8807bf80a280 000000018108265b ffff8807a0c03c00 0000000000000001
> [ 2144.673300] Call Trace:
> [ 2144.673302]  <IRQ>
> [ 2144.673305]  [<ffffffff81086713>] load_balance+0x543/0x630
> [ 2144.673309]  [<ffffffff8108689d>] rebalance_domains+0x9d/0x180
> [ 2144.673311]  [<ffffffff81086b44>] run_rebalance_domains+0x44/0x60
> [ 2144.673315]  [<ffffffff81050376>] __do_softirq+0xd6/0x250
> [ 2144.673318]  [<ffffffff810505f5>] irq_exit+0xb5/0xc0
> [ 2144.673322]  [<ffffffff81027b89>] smp_apic_timer_interrupt+0x69/0xa0
> [ 2144.673325]  [<ffffffff814aa09d>] apic_timer_interrupt+0x6d/0x80
> [ 2144.673327]  <EOI>
> [ 2144.673329]  [<ffffffff814a10d6>] ? _raw_spin_lock+0x66/0x80
> [ 2144.673331]  [<ffffffff81170596>] path_get+0x26/0x40
> [ 2144.673334]  [<ffffffff8117070a>] unlazy_walk+0x10a/0x230
> [ 2144.673337]  [<ffffffff81170a59>] lookup_fast+0x229/0x2d0
> [ 2144.673340]  [<ffffffff81172e13>] path_lookupat+0x123/0x720
> [ 2144.673342]  [<ffffffff8116f663>] ? inode_permission+0x13/0x50
> [ 2144.673344]  [<ffffffff81172438>] ? link_path_walk+0x78/0x450
> [ 2144.673434]  [<ffffffff8117343f>] filename_lookup+0x2f/0xc0
> [ 2144.673438]  [<ffffffff811747d4>] user_path_at_empty+0x54/0xa0
> [ 2144.673441]  [<ffffffff8105c731>] ? group_send_sig_info+0x21/0x60
> [ 2144.673444]  [<ffffffff8105c7aa>] ? kill_pid_info+0x3a/0x60
> [ 2144.673523]  [<ffffffff8117482c>] user_path_at+0xc/0x10
> [ 2144.673529]  [<ffffffff81169c01>] vfs_fstatat+0x51/0xb0
> [ 2144.673532]  [<ffffffff81169d36>] vfs_stat+0x16/0x20
> [ 2144.673534]  [<ffffffff81169d5f>] sys_newstat+0x1f/0x50
> [ 2144.673538]  [<ffffffff810c8af6>] ? __audit_syscall_exit+0x246/0x2f0
> [ 2144.673541]  [<ffffffff810c79bc>] ? __audit_syscall_entry+0x8c/0xf0
> [ 2144.673543]  [<ffffffff814a9429>] system_call_fastpath+0x16/0x1b

This is on a 40 core / 80 thread Westmere-EX with 16 VMs, each VM having
20 vCPUs (so 4x over-commit).  All VMs run dbench in tmpfs, which is a
pretty good test for spinlock preempt problems.  I had PLE enabled for
the test.

When you re-base your patches I will try it again.

Thanks,

-Andrew Theurer