From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BYAPR05CU005.outbound.protection.outlook.com (mail-westusazon11010052.outbound.protection.outlook.com [52.101.85.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8DAC3BA24E for ; Wed, 29 Apr 2026 09:00:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.85.52 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777453225; cv=fail; b=iirDsa32cj3TD3OBCovtJq1QqH7+ldANnDgwMP50B3s1InNeo2DB5ffhdYnLScDnKNi5me7jQ9i1eddxodJz86AJpygYbLF6sFPmWxnI/7Q7AlENc9VKc158T7ZnulIQmXIkfXQKp2kzsl6FMKiv2NSmaxm/MO6AyRkFzD9u1YU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777453225; c=relaxed/simple; bh=a6cQAo8Mp1/b2rzZBXfMute8xwoXZjW1W7JF7rXCu64=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=D+1CDkn45v4AI0ncHPoLqQJnxw+eiNCOa3ldZDNSSVTn78Q6uxB35LpA7a36VM/oL4IWB5Cx5hWWuXGJMMytpEMYjmkagvCqb3kSbEvcvx30sO7gEOKBSgP4L1LMr7SdXG1fzqYVs0hS8kqf+fPuiChXTF2wBlaL+We1ynpoOyM= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=SavOSFqN; arc=fail smtp.client-ip=52.101.85.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="SavOSFqN" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Ry3GMOpEIMkAARFsYzoE9bUur0PkKtwKcQ7FOpsMZlR7wSdf9QL1JsnRqRHYNbeoPiirmfQ6CC78j/g+lYdhAgIsiuilYTgsh7Wu3aiwpC9RzPBbJwY/nnbKiiUQmxcJfB3QqPspxg6XDo99rS1kcIgMFVQPITFab6LVbbEDgzqs5ljQ6rlXYwEkbZ6O3hnMDmJZ9xoOl0+uHp1pyfeHmdjmfgEgrljGs7nOqJqiaBab+0xNXO1GNU5VRpR6be+GPvEru0vJLGKAs5kqERdbjDdu3tUQUyZnjo1B/WMWsDc/A8Z/Zjd24WCBaSD+3uOpgByzenFCDumly33IWJ2plw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5TWrwRk9Xw9ta5aR+ziwWQHZfxoGYpdebNB37/8Njtk=; b=kuubdrQVUFJnk/11g8fUJj+O1tfJC6FogDwsVUSIY0LRYtef1g4VU+P45hte6QpeTUTbUb4fBq8YBnKB0tE8KLIpVHay8g1AbWjjqU0rZ8IiMvdlbUF9uua4jYfeBUrgih7fIZctj1FFfak/uVmNps+BNVx04pKLOniJG3jOu+oXgjMNX+AqOGQvyW/ybArWLaW+Y8fjcZslOJNgEW4C/KUR/qSGWmCv+5mUq3ysg5sSBN0TxgVfihC8V1uuKZc8TpQYGlxti6wiTGv65ywwOsizzmG4WTykszFTqzk8f3u7bw0EnrFwlXiXpohg8eOUaFifvxo9KajKlSk38qnhzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=google.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5TWrwRk9Xw9ta5aR+ziwWQHZfxoGYpdebNB37/8Njtk=; b=SavOSFqN9kMb+HfHE34K4w2sbJ+5GW9MQ1gslpuPI/0ydIyCn5810bt3nBWZPxY5Q8dcWs0WQTNKQ/web4M6vCAyRWCcr5DbR5cPDVnu1PiRhaI4z7xmFie5YxQTaUmqnLTiFdFSQUU/RYGihdkeIO4X+jrJ5M04vR7TQzxzenI= Received: from CH3P221CA0002.NAMP221.PROD.OUTLOOK.COM (2603:10b6:610:1e7::25) by IA1PR12MB6164.namprd12.prod.outlook.com (2603:10b6:208:3e8::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.17; Wed, 29 Apr 2026 09:00:17 +0000 Received: from CH1PEPF0000A349.namprd04.prod.outlook.com (2603:10b6:610:1e7:cafe::ce) by CH3P221CA0002.outlook.office365.com (2603:10b6:610:1e7::25) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9846.30 via Frontend Transport; Wed, 29 Apr 2026 09:00:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb08.amd.com; pr=C Received: from satlexmb08.amd.com (165.204.84.17) by CH1PEPF0000A349.mail.protection.outlook.com (10.167.244.9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.18 via Frontend Transport; Wed, 29 Apr 2026 09:00:17 +0000 Received: from Satlexmb09.amd.com (10.181.42.218) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Wed, 29 Apr 2026 04:00:06 -0500 Received: from satlexmb08.amd.com (10.181.42.217) by satlexmb09.amd.com (10.181.42.218) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Wed, 29 Apr 2026 02:00:06 -0700 Received: from [10.136.41.76] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Wed, 29 Apr 2026 03:59:59 -0500 Message-ID: Date: Wed, 29 Apr 2026 14:29:58 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/2] sched: proxy-exec: Close race causing workqueue work being delayed To: John Stultz CC: Peter Zijlstra , LKML , Vineeth Pillai , "Sonam Sanju" , Sean Christopherson , "Kunwu Chan" , Tejun Heo , Joel Fernandes , Qais Yousef , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Will Deacon , Waiman Long , Boqun Feng , "Paul E. McKenney" , Metin Kaya , Xuewen Yan , Thomas Gleixner , "Daniel Lezcano" , Suleiman Souhlal , kuyo chang , hupu , References: <20260427183848.698551-1-jstultz@google.com> <20260427183848.698551-2-jstultz@google.com> <20260428094353.GB1026330@noisy.programming.kicks-ass.net> <20260428111833.GL3102924@noisy.programming.kicks-ass.net> <9cf9b433-cba5-4a8e-8dbf-6410239cffb6@amd.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH1PEPF0000A349:EE_|IA1PR12MB6164:EE_ X-MS-Office365-Filtering-Correlation-Id: d2e49093-0ed1-4b4a-0290-08dea5cdbc1d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|7416014|376014|82310400026|1800799024|22082099003|56012099003|18002099003; X-Microsoft-Antispam-Message-Info: GU6VGxgdabR6135qORTMqVGViumaUpyEDEdTvL+BjuuZW3YhygRUedR5/pv6dXuofYlR0r+MniRa7lzKC7LP1SChzxN93VUL1cF7KweyMZZh4rctS5+WkGP3fwSY301QrLYD5tywJGxnd3cQprt9tzCDVB3xIku3tD8oyX/K35TzLjBvI8dP7Meu39nuOZci+IBN4gPQbcBQUbPT1KpiBvXxPJgIkSAjI4LYUYZHUuCbm25ZUev4s68Ov1Dw20hEv8VMqMsBueQ87iqiRZlGsmtT3XQfa5dR0zgIwTNd/DSjj00dceJ8yE1wyI4iuQTVjTMH1mM4DvM0JQ++hqSZdPweD30layVI5oFvNn/cq09HD+0o1oaTPVpV6ndvrcRiPpu15GF+sSep/5Xf4lsEzgURjO2tlMdJl67G+zqMTDns3YOROZbHAJB0PVa+qo5C/6UNL34TGE57sqICWXuvmXZzdEaStv5MIu1/3shYKHn6CIzm4YDv9MZ0YEu8Y8k6FxtVl04qVlR4cSw6e6TvSdaWn2LlHWsDQkoGYdVmgchBRev06HE3n8Kij23rtsjLQmSPAPflNWYBYALaJK3A8aaUL0OkYG7GvPeXJxRVh5GYvUtYXiIIfio7OUZQw28KE0KLERcHGUcZS8tgeRtN25nfq25n+EbzAVn/lMvrR7yMdw9KrBxuy0haFHEPSc6DaCKVpFpFY4a1iF497r3hJY0WCz2D8vd/lGhR/N2RvUAsG689tsobONO1Jtmv7a1DcLxDqbAPFWrGtoEcnaCJEw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb08.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(7416014)(376014)(82310400026)(1800799024)(22082099003)(56012099003)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: A0dOuQWb5kqTC+2TBLJwp0p1GjVZsGbBrzyHANVT3nqvOxPFIJqWnXnEkdcW21zJa93RHa9LpzCSdc8jYKO8yMtYXmYzLnaTySDfC42oZvnnI/Ji1pMQGju8WvYcfAtadhwXLAzcmfAE7jsZD0koXyq3ztrWMM/GarbkxE5tJNFs5Wv4011f9Geu5Ksn1VRb4ajeN5HHFOhtfd029CU3PNfrRgf3duBY88VPVPBNm3VW+YNJOu64GqyZYBZiaTbs4OcwF/mCrEg/4Jof83aG0GwhiHGumMrMV09YwegfGZlYgO4pcKDkzxYUlpfi5lujZGyDVkUCBlYYJq+QIodHLTw1AGv5qTS90D9a1Yc9raKRYAjhpibtL65jDv4XL/jcu/VWig/MIRS7qvXYUKKFL2goY6faCP8/JJVabpGavu1Lx7+Vtpc1COPdNsjqwJw3 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Apr 2026 09:00:17.5535 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: d2e49093-0ed1-4b4a-0290-08dea5cdbc1d X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb08.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CH1PEPF0000A349.namprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6164 Hello John, On 4/29/2026 7:57 AM, John Stultz wrote: >> diff --git a/include/linux/sched.h b/include/linux/sched.h >> index 8ec3b6d7d718b..6ea74aecc5fbd 100644 >> --- a/include/linux/sched.h >> +++ b/include/linux/sched.h >> @@ -586,6 +586,7 @@ struct sched_entity { >> unsigned char sched_delayed; >> unsigned char rel_deadline; >> unsigned char custom_slice; >> + unsigned char sched_proxy; >> /* hole */ > > I feel like this is so tied to the blocked_on value, I suspect it > makes the most sense to have this flag be the low bit of that pointer? > > Sort of a blocked_on latch, to signal its really in effect? > > Plus it gets cleared automatically on set and clear, so it looks a > little cleaner. I had actually started looking into that last night but sleep eventually got to me! I agree that is much cleaner to do it alongside p->blocked_on. > > >> @@ -6535,8 +6536,13 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p, >> * blocked on a mutex, and we want to keep it on the runqueue >> * to be selectable for proxy-execution. >> */ >> - if (!should_block) >> + if (!should_block) { >> + guard(raw_spinlock)(&p->blocked_lock); >> + /* Stable against race */ >> + if (task_is_blocked(p)) >> + WRITE_ONCE(p->se.sched_proxy, 1); >> return false; >> + } > > So if we double check and find the task isn't blocked anymore, we > probably shouldn't return early here, no? > > Let me take a stab at the bit flag approach and see how it goes. In case you want to peek at my homework ;-) (Lightly tested with test-ww_mutex and sched-messaging) diff --git a/include/linux/sched.h b/include/linux/sched.h index 8ec3b6d7d718b..e28f5c05a1689 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1237,7 +1237,7 @@ struct task_struct { struct rt_mutex_waiter *pi_blocked_on; #endif - struct mutex *blocked_on; /* lock we're blocked on */ + unsigned long blocked_on; /* lock we're blocked on */ raw_spinlock_t blocked_lock; #ifdef CONFIG_DETECT_HUNG_TASK_BLOCKER @@ -2188,12 +2188,20 @@ extern int __cond_resched_rwlock_write(rwlock_t *lock) __must_hold(lock); * evaluated for return-migration before it is run. So if the task is * blocked_on PROXY_WAKING, return migrate it before running it. */ -#define PROXY_WAKING ((struct mutex *)(-1L)) +#define PROXY_WAKING ((unsigned long)(-1L)) + +#define BO_FLAG_PROXY BIT(0) +#define BO_FLAGS GENMASK(0, 0) + +static inline struct mutex *__get_blocked_on_mutex(unsigned long m) +{ + return (struct mutex *)(m & ~BO_FLAGS); +} static inline struct mutex *__get_task_blocked_on(struct task_struct *p) { lockdep_assert_held_once(&p->blocked_lock); - return p->blocked_on == PROXY_WAKING ? NULL : p->blocked_on; + return p->blocked_on == PROXY_WAKING ? NULL : __get_blocked_on_mutex(p->blocked_on); } static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m) @@ -2208,8 +2216,11 @@ static inline void __set_task_blocked_on(struct task_struct *p, struct mutex *m) * with a different mutex. Note, setting it to the same * lock repeatedly is ok. */ - WARN_ON_ONCE(p->blocked_on && p->blocked_on != m); - p->blocked_on = m; + WARN_ON_ONCE(p->blocked_on && + __get_blocked_on_mutex(p->blocked_on) != m); + + BUG_ON((unsigned long)m & BO_FLAGS); + p->blocked_on = (unsigned long)m & ~BO_FLAGS; } static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex *m) @@ -2221,8 +2232,10 @@ static inline void __clear_task_blocked_on(struct task_struct *p, struct mutex * * blocked_on relationships, but make sure we are not * clearing the relationship with a different lock. */ - WARN_ON_ONCE(m && p->blocked_on && p->blocked_on != m && p->blocked_on != PROXY_WAKING); - p->blocked_on = NULL; + WARN_ON_ONCE(m && p->blocked_on && + __get_blocked_on_mutex(p->blocked_on) != m && + p->blocked_on != PROXY_WAKING); + p->blocked_on = 0; } static inline void clear_task_blocked_on(struct task_struct *p, struct mutex *m) @@ -2249,7 +2262,12 @@ static inline void __set_task_blocked_on_waking(struct task_struct *p, struct mu * already set to waking, but make sure we are not changing * the relationship with a different lock. */ - WARN_ON_ONCE(m && p->blocked_on != m && p->blocked_on != PROXY_WAKING); + WARN_ON_ONCE(m && + __get_blocked_on_mutex(p->blocked_on) != m && + p->blocked_on != PROXY_WAKING); + + /* Always ensure PROXY_WAKING implies BO_FLAG_PROXY. */ + BUILD_BUG_ON(!(PROXY_WAKING & BO_FLAG_PROXY)); p->blocked_on = PROXY_WAKING; } @@ -2259,6 +2277,20 @@ static inline void set_task_blocked_on_waking(struct task_struct *p, struct mute __set_task_blocked_on_waking(p, m); } +static inline void set_task_blocked_on_proxy(struct task_struct *p) +{ + guard(raw_spinlock_irqsave)(&p->blocked_lock); + + if (!sched_proxy_exec()) + return; + + /* Don't add BO_FLAG_PROXY if blocked_on was already cleared */ + if (!p->blocked_on) + return; + + p->blocked_on |= BO_FLAG_PROXY; +} + #else static inline void __clear_task_blocked_on(struct task_struct *p, struct rt_mutex *m) { diff --git a/kernel/fork.c b/kernel/fork.c index 079802cb61002..936e8d6ee7a0b 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2177,7 +2177,7 @@ __latent_entropy struct task_struct *copy_process( lockdep_init_task(p); - p->blocked_on = NULL; /* not blocked yet */ + p->blocked_on = 0; /* not blocked yet */ #ifdef CONFIG_BCACHE p->sequential_io = 0; diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 49cd5d2171613..a2d61ec40a421 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6535,8 +6535,10 @@ static bool try_to_block_task(struct rq *rq, struct task_struct *p, * blocked on a mutex, and we want to keep it on the runqueue * to be selectable for proxy-execution. */ - if (!should_block) + if (!should_block) { + set_task_blocked_on_proxy(p); return false; + } p->sched_contributes_to_load = (task_state & TASK_UNINTERRUPTIBLE) && @@ -6765,18 +6767,24 @@ find_proxy_task(struct rq *rq, struct task_struct *donor, struct rq_flags *rf) bool curr_in_chain = false; int this_cpu = cpu_of(rq); struct task_struct *p; - struct mutex *mutex; int owner_cpu; /* Follow blocked_on chain. */ - for (p = donor; (mutex = p->blocked_on); p = owner) { - /* if its PROXY_WAKING, do return migration or run if current */ - if (mutex == PROXY_WAKING) { - if (task_current(rq, p)) { - clear_task_blocked_on(p, PROXY_WAKING); - return p; + for (p = donor; task_is_blocked(p); p = owner) { + struct mutex *mutex = __get_task_blocked_on(p); + + if (!mutex) { + /* if its PROXY_WAKING, do return migration or run if current */ + if (p->blocked_on == PROXY_WAKING) { + if (task_current(rq, p)) { + clear_task_blocked_on(p, (struct mutex *)PROXY_WAKING); + return p; + } + goto force_return; } - goto force_return; + + /* Something changed in the blocked_on chain; Try again. */ + return NULL; } /* @@ -7026,13 +7034,15 @@ static void __sched notrace __schedule(int sched_mode) } } else if (!preempt && prev_state) { /* - * We pass task_is_blocked() as the should_block arg + * We pass prev->blocked_on as the should_block arg * in order to keep mutex-blocked tasks on the runqueue * for slection with proxy-exec (without proxy-exec * task_is_blocked() will always be false). + * + * task_is_blocked() cannot be used directly since the + * task has not set BO_FLAG_PROXY yet. */ - try_to_block_task(rq, prev, &prev_state, - !task_is_blocked(prev)); + try_to_block_task(rq, prev, &prev_state, !prev->blocked_on); switch_count = &prev->nvcsw; } @@ -7043,8 +7053,16 @@ static void __sched notrace __schedule(int sched_mode) if (sched_proxy_exec()) { struct task_struct *prev_donor = rq->donor; + /* + * A wakeup raced with block_task(); + * Clear blocked_on before running the task + * again. + */ + if (unlikely(!prev_state && prev->blocked_on)) + clear_task_blocked_on(prev, NULL); + rq_set_donor(rq, next); - if (unlikely(next->blocked_on)) { + if (unlikely(task_is_blocked(next))) { next = find_proxy_task(rq, next, &rf); if (!next) { zap_balance_callbacks(rq); diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index c95584191d58f..e4a0aeee9bfc0 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -2390,7 +2390,7 @@ static inline bool task_is_blocked(struct task_struct *p) if (!sched_proxy_exec()) return false; - return !!p->blocked_on; + return !!(p->blocked_on & BO_FLAG_PROXY); } static inline int task_on_cpu(struct rq *rq, struct task_struct *p) --- There is probably a better way to do it by inspecting the flags only in find_proxy_task() and keep task_is_blocked() as is but I wanted to see if it can be done as a part of task_ib_blocked() helper since that seemed cleaner in my head. I'll let you fiddle with it. -- Thanks and Regards, Prateek