From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756550AbdLVXKO (ORCPT ); Fri, 22 Dec 2017 18:10:14 -0500 Received: from mx2.suse.de ([195.135.220.15]:38035 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752207AbdLVXKM (ORCPT ); Fri, 22 Dec 2017 18:10:12 -0500 X-Amavis-Alert: BAD HEADER SECTION, Duplicate header field: "Cc" From: NeilBrown To: Peter Zijlstra Date: Sat, 23 Dec 2017 10:10:02 +1100 Cc: Ingo Molnar , linux-kernel@vger.kernel.org Cc: Oleg Drokin , Andreas Dilger , James Simmons , Patrick Farrell Subject: Re: [PATCH 2/2] sched/wait: add wait_event_idle_exclusive_lifo() In-Reply-To: <20171222080841.22kptq7gzmvswfvk@hirez.programming.kicks-ass.net> References: <151391203805.23157.8332034037737592042.stgit@noble> <151391226407.23157.14719462856574834182.stgit@noble> <20171222080841.22kptq7gzmvswfvk@hirez.programming.kicks-ass.net> Message-ID: <8737428ilx.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="=-=-="; micalg=pgp-sha256; protocol="application/pgp-signature" Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org --=-=-= Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, Dec 22 2017, Peter Zijlstra wrote: > On Fri, Dec 22, 2017 at 02:11:04PM +1100, NeilBrown wrote: >> wait_event_*_exclusive() adds new waiters to the end of the >> quest, while non-exclusive wait_event adds to the head. >>=20 >> This ensures that a wake_up will wake all non-exclusive >> waiters and at most one exclusive wait, but it means that >> exclusive waiters are woken in a FIFO order, so the task >> woken is the one least likely to have data in the CPU cache. >>=20 >> When simple interaction with non-exclusive waiters is not >> important, and when choosing a cache-hot task is, the new >>=20 >> wait_event_idle_exclusive_lifo() >> and >> wait_event_idle_exclusive_lifo_timeout() >>=20 >> can be used. To implement these we introduce a new >> WQ_FLAG_LIFO which causes prepare_to_wait_event() to >> add to the head of the queue. >>=20 >> This will be used to allow lustre's l_wait_event() to be >> replaced with more standard wait.h macros. > > Urgh, so the problem with lifo is that it tends to generate starvation > so you have to be very careful with how one uses it. True, but given that starvation is key to the design goal here, that doesn't seem like an argument against it. If we can starve some processes, they get thinner (in terms of CPU cache usage) and so impose less burden... All processes doing the exclusive_lifo wait are effectively equal, so if several are idle it doesn't matter at all for correctness which is woken. This would be an argument against a wait_event* version that uses TASK_UNINTERRUPTIBLE, but not against one that uses TASK_IDLE. > > Is there really a measurable difference if you make lustre use the > regular fifo stuff? The only background I know is a commit from git://git.hpdd.intel.com/fs/lustre-dev.git which says: =2D-------- commit 40e312a8275ed9240e63f0ac023d8b7a38136f42 Author: Jian Yu Date: Wed Dec 1 20:16:21 2010 +0800 b=3D23289 new API: cfs_waitq_add_exclusive_head =20=20=20=20 With this patch, we can reduce total number of active threads because waitq is a LIFO list for exclusive waiting. =20=20=20=20 o=3DLiang Zhen i=3Dandreas.dilger i=3Deric.mei =2D--------- Maybe someone more familiar with lustre history could help?? > >> Signed-off-by: NeilBrown >> --- >> include/linux/wait.h | 95 +++++++++++++++++++++++++++++++++++++++++++= ++++--- >> kernel/sched/wait.c | 3 +- >> 2 files changed, 91 insertions(+), 7 deletions(-) >>=20 >> diff --git a/include/linux/wait.h b/include/linux/wait.h >> index 3aea0780c9d0..49cb393c53d5 100644 >> --- a/include/linux/wait.h >> +++ b/include/linux/wait.h >> @@ -20,6 +20,9 @@ int default_wake_function(struct wait_queue_entry *wq_= entry, unsigned mode, int >> #define WQ_FLAG_EXCLUSIVE 0x01 >> #define WQ_FLAG_WOKEN 0x02 >> #define WQ_FLAG_BOOKMARK 0x04 >> +#define WQ_FLAG_LIFO 0x08 /* used with WQ_FLAG_EXCLUSIVE to force >> + * LIFO scheduling in prepare_to_wait_event(). >> + */ > > This is not an acceptable comment style. Fixed to /* * WQ_FLAG_LIFO is used with WQ_FLAG_EXCLUSIVE * to force LIFO scheduling in prepare_to_wait_event(). */ #define WQ_FLAG_LIFO 0x08 Thanks, NeilBrown --=-=-= Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEG8Yp69OQ2HB7X0l6Oeye3VZigbkFAlo9kMsACgkQOeye3VZi gbm6CRAAnKmtImG7Y4VOfEdolrHNCvcmts9UKPIG9leEckoeojq4JZEzJjgajiSp RYrrtacrlqDeVXxCQYeWbv9ssRpoOoqs5xKGsHk1OT6bh+tMb45TW+RsC5Fs4lzW LsbdAawJ7DWzfoHJ78HCiGWyrWuhYUHVH28OgOuGvrl4ucMTvg37i9zCXnURHLou Fy8XDIvbseKjnXeorDAAKamoPqJrjo/SXhlChY9iINOyPkJuRX3TDQngErBKWCTa 3CLKc6LpgCkMKuBJOVnVSZv9oxH9Z3u94/x4zRHOL/mtuy9OHtjI1Kvs8j6HWCD5 RwFnCj0nFmbZdLfeR4zIsjXTcgEUxAOHxYweTpQBE7UyGQpUpm4WLh6ljvVPQPhZ w0SL6PQ+w1ly3Br0vyKlj5ppAJb1g+O1bPFfX/XUzQ+XuMehto1jPYUpFVGo6bin sFK8DTMs1CzpxQJ8uB2Za4E4QOl7OaxO7nDWbLQ6wR7KBiaX9957KosmnbssllLX X4Ss3zgTotj8lNAF/r872+jk15edu0N/flziz4GKBxSjlaWV3sAK/6FVTjtJMDw2 KbGpoAhG7al32FDn3z6Y3gtnYKHPV77YMvf7mfwKYBY63tKtVWUPDFBBEIr5Ralt 3iaCkpClSHlmP0NTzZAk4lAxHS1NE76AagnoZKq4zrYWdL1ERhM= =cKW0 -----END PGP SIGNATURE----- --=-=-=--