[PATCH 0/4] test-ww_mutex: Miscellaneous fixes

[PATCH 0/4] test-ww_mutex: Miscellaneous fixes

[Håkon Bugge]

This series fixes:
     - module cleanup
     - error propagation from stress workers
     - transient -EDEADLK handling in test_cycle_work()
     - timeout-related false negatives in stress_inorder_work()

The series is based on v7.1 and has been tested on a 96-CPU x86_64
bare-metal system and a 160-CPU arm64 bare-metal system. Both systems
completed 100 iterations of
tools/testing/selftests/locking/ww_mutex.sh successfully.

It is worth mentioning that the last patch retries up to 10,000 lock
acquisitions after timeout. This value was determined empirically: 10
and 100 retries were insufficient on the test systems, while 10,000
provided stable results and avoided false negatives.

Håkon Bugge (4):
  test-ww_mutex: Fix module cleanup
  test-ww_mutex: Report errors from stress workers
  test-ww_mutex: Handle transient -EDEADLK in test_cycle_work
  test-ww_mutex: Retry lock acquisition after timeout if deadlocked

 kernel/locking/test-ww_mutex.c | 55 +++++++++++++++++++++++++---------
 1 file changed, 41 insertions(+), 14 deletions(-)

--
2.43.5

[PATCH 1/4] test-ww_mutex: Fix module cleanup

[Håkon Bugge]

When test-ww_mutex is compiled as a module, there are two bugs. One is
related to the case where the module fails to load. In that case, some
or all of the allocated objects are not released. Also, if loaded OK
and thereafter unloaded, the sysfs file is not removed.

The latter bug gives the following splat (trimmed for better brevity):

sysfs: cannot create duplicate filename '/kernel/test_ww_mutex'
 dump_stack_lvl+0x5d/0x90
 sysfs_warn_dup.cold+0x17/0x23
 sysfs_create_dir_ns+0xe2/0x100
 kobject_add_internal+0xba/0x270
 kobject_add+0x95/0xd0
 kobject_create_and_add+0xa3/0xd0
 test_ww_mutex_init+0x90/0xff0 [test_ww_mutex]
 do_one_initcall+0x5c/0x330
 do_init_module+0x90/0x290
 __do_sys_init_module+0x195/0x1d0
 do_syscall_64+0xe8/0x550
 entry_SYSCALL_64_after_hwframe+0x76/0x7e

Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
Fixes: de2c5a1523fd ("test-ww_mutex: Allow test to be run (and re-run) from userland")
Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
---
 kernel/locking/test-ww_mutex.c | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
index 838d631544ed2..6f56a29ed6d77 100644
--- a/kernel/locking/test-ww_mutex.c
+++ b/kernel/locking/test-ww_mutex.c
@@ -734,7 +734,7 @@ static struct kobject *test_ww_mutex_kobj;
 
 static int __init test_ww_mutex_init(void)
 {
-	int ret;
+	int ret = -ENOMEM;
 
 	prandom_seed_state(&rng, get_random_u64());
 
@@ -743,28 +743,35 @@ static int __init test_ww_mutex_init(void)
 		return -ENOMEM;
 
 	test_ww_mutex_kobj = kobject_create_and_add("test_ww_mutex", kernel_kobj);
-	if (!test_ww_mutex_kobj) {
-		destroy_workqueue(wq);
-		return -ENOMEM;
-	}
+	if (!test_ww_mutex_kobj)
+		goto kobj_err;
 
 	/* Create the files associated with this kobject */
 	ret = sysfs_create_group(test_ww_mutex_kobj, &attr_group);
-	if (ret) {
-		kobject_put(test_ww_mutex_kobj);
-		destroy_workqueue(wq);
-		return ret;
-	}
+	if (ret)
+		goto sysfs_err;
 
 	mutex_lock(&run_lock);
 	ret = run_test_classes();
 	mutex_unlock(&run_lock);
 
+	if (!ret)
+		return 0;
+
+	sysfs_remove_group(test_ww_mutex_kobj, &attr_group);
+
+sysfs_err:
+	kobject_put(test_ww_mutex_kobj);
+
+kobj_err:
+	destroy_workqueue(wq);
+
 	return ret;
 }
 
 static void __exit test_ww_mutex_exit(void)
 {
+	sysfs_remove_group(test_ww_mutex_kobj, &attr_group);
 	kobject_put(test_ww_mutex_kobj);
 	destroy_workqueue(wq);
 }
-- 
2.43.5

Re: [PATCH 1/4] test-ww_mutex: Fix module cleanup

[John Stultz]

On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>
> When test-ww_mutex is compiled as a module, there are two bugs. One is
> related to the case where the module fails to load. In that case, some
> or all of the allocated objects are not released. Also, if loaded OK
> and thereafter unloaded, the sysfs file is not removed.
>
> The latter bug gives the following splat (trimmed for better brevity):
>
> sysfs: cannot create duplicate filename '/kernel/test_ww_mutex'
>  dump_stack_lvl+0x5d/0x90
>  sysfs_warn_dup.cold+0x17/0x23
>  sysfs_create_dir_ns+0xe2/0x100
>  kobject_add_internal+0xba/0x270
>  kobject_add+0x95/0xd0
>  kobject_create_and_add+0xa3/0xd0
>  test_ww_mutex_init+0x90/0xff0 [test_ww_mutex]
>  do_one_initcall+0x5c/0x330
>  do_init_module+0x90/0x290
>  __do_sys_init_module+0x195/0x1d0
>  do_syscall_64+0xe8/0x550
>  entry_SYSCALL_64_after_hwframe+0x76/0x7e
>
> Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
> Fixes: de2c5a1523fd ("test-ww_mutex: Allow test to be run (and re-run) from userland")
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>

Ah! Thanks for sending this!

Acked-by: John Stultz <jstultz@google.com>

[PATCH 2/4] test-ww_mutex: Report errors from stress workers

[Håkon Bugge]

If any of stress_one_work(), stress_reorder_work(), or
stress_inorder_work() fails, the error is not conveyed back to the
test harness function stress().

We may then see in the log:

   stress (stress_inorder_work) failed with -35
   Beginning ww (die) mutex selftests
   All ww mutex selftests passed

Besides being a contradiction by terms, the failure to return error
from said worker functions leads to false-positive results.

We fix that by adding a *result* variable to *struct stress*, update
it in said worker functions, and finally, in stress(), scan the
*stress_array* and report the first error if any.

Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
---
 kernel/locking/test-ww_mutex.c | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
index 6f56a29ed6d77..5a4c92801bdfb 100644
--- a/kernel/locking/test-ww_mutex.c
+++ b/kernel/locking/test-ww_mutex.c
@@ -392,6 +392,7 @@ struct stress {
 	struct ww_class *class;
 	unsigned long timeout;
 	int nlocks;
+	int result;
 };
 
 struct rnd_state rng;
@@ -440,6 +441,9 @@ static void stress_inorder_work(struct work_struct *work)
 	struct ww_mutex *locks = stress->locks;
 	struct ww_acquire_ctx ctx;
 	int *order;
+	int err;
+
+	stress->result = -ENOMEM;
 
 	order = get_random_order(nlocks);
 	if (!order)
@@ -447,7 +451,7 @@ static void stress_inorder_work(struct work_struct *work)
 
 	do {
 		int contended = -1;
-		int n, err;
+		int n;
 
 		ww_acquire_init(&ctx, stress->class);
 retry:
@@ -485,6 +489,7 @@ static void stress_inorder_work(struct work_struct *work)
 	} while (!time_after(jiffies, stress->timeout));
 
 	kfree(order);
+	stress->result = err;
 }
 
 struct reorder_lock {
@@ -501,6 +506,8 @@ static void stress_reorder_work(struct work_struct *work)
 	int *order;
 	int n, err;
 
+	stress->result = -ENOMEM;
+
 	order = get_random_order(stress->nlocks);
 	if (!order)
 		return;
@@ -545,6 +552,8 @@ static void stress_reorder_work(struct work_struct *work)
 		ww_acquire_fini(&ctx);
 	} while (!time_after(jiffies, stress->timeout));
 
+	stress->result = err;
+
 out:
 	list_for_each_entry_safe(ll, ln, &locks, link)
 		kfree(ll);
@@ -569,6 +578,8 @@ static void stress_one_work(struct work_struct *work)
 			break;
 		}
 	} while (!time_after(jiffies, stress->timeout));
+
+	stress->result = err;
 }
 
 #define STRESS_INORDER BIT(0)
@@ -581,6 +592,7 @@ static int stress(struct ww_class *class, int nlocks, int nthreads, unsigned int
 	struct ww_mutex *locks;
 	struct stress *stress_array;
 	int n, count;
+	int res = 0;
 
 	locks = kmalloc_objs(*locks, nlocks);
 	if (!locks)
@@ -635,10 +647,15 @@ static int stress(struct ww_class *class, int nlocks, int nthreads, unsigned int
 
 	for (n = 0; n < nlocks; n++)
 		ww_mutex_destroy(&locks[n]);
+
+	/* Report the first error */
+	for (n = 0; !res && n < count; n++)
+		res = stress_array[n].result;
+
 	kfree(stress_array);
 	kfree(locks);
 
-	return 0;
+	return res;
 }
 
 static int run_tests(struct ww_class *class)
-- 
2.43.5

Re: [PATCH 2/4] test-ww_mutex: Report errors from stress workers

[John Stultz]

On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>
> If any of stress_one_work(), stress_reorder_work(), or
> stress_inorder_work() fails, the error is not conveyed back to the
> test harness function stress().
>
> We may then see in the log:
>
>    stress (stress_inorder_work) failed with -35
>    Beginning ww (die) mutex selftests
>    All ww mutex selftests passed
>
> Besides being a contradiction by terms, the failure to return error
> from said worker functions leads to false-positive results.
>
> We fix that by adding a *result* variable to *struct stress*, update
> it in said worker functions, and finally, in stress(), scan the
> *stress_array* and report the first error if any.
>
> Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>

Acked-by: John Stultz <jstultz@google.com>

Re: [PATCH 2/4] test-ww_mutex: Report errors from stress workers

[John Stultz]

On Thu, Jun 18, 2026 at 10:52 AM John Stultz <jstultz@google.com> wrote:
> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
> >
> > If any of stress_one_work(), stress_reorder_work(), or
> > stress_inorder_work() fails, the error is not conveyed back to the
> > test harness function stress().
> >
> > We may then see in the log:
> >
> >    stress (stress_inorder_work) failed with -35
> >    Beginning ww (die) mutex selftests
> >    All ww mutex selftests passed
> >
> > Besides being a contradiction by terms, the failure to return error
> > from said worker functions leads to false-positive results.
> >
> > We fix that by adding a *result* variable to *struct stress*, update
> > it in said worker functions, and finally, in stress(), scan the
> > *stress_array* and report the first error if any.
> >
> > Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
> > Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>
> Acked-by: John Stultz <jstultz@google.com>

I did catch one issue with this patch in testing.  Where as without
this change, if there was a failure in the first run during
test_ww_mutex_init(), you could still run the test again manually via
the sysfs interface without unloading and reloading.

But with this change, since the error is reported, the module_init
function sees the error and bails out, removing the sysfs interface.
If its a module, you can just reload it, but if it is built-in then
its stuck and can't be rerun.

So we might want to tweak test_ww_mutex_init() with this patch and
include something like (whitespace corrupted, just for illustration
here):
        ret = run_test_classes();
        mutex_unlock(&run_lock);

-       if (!ret)
+       if (!ret || IS_BUILTIN(CONFIG_WW_MUTEX_SELFTEST))
                return 0;

        sysfs_remove_group(test_ww_mutex_kobj, &attr_group);

?

thanks
-john

Re: [PATCH 2/4] test-ww_mutex: Report errors from stress workers

[Haakon Bugge]

> On 9 Jul 2026, at 02:03, John Stultz <jstultz@google.com> wrote:
> 
> On Thu, Jun 18, 2026 at 10:52 AM John Stultz <jstultz@google.com> wrote:
>> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>>> 
>>> If any of stress_one_work(), stress_reorder_work(), or
>>> stress_inorder_work() fails, the error is not conveyed back to the
>>> test harness function stress().
>>> 
>>> We may then see in the log:
>>> 
>>>   stress (stress_inorder_work) failed with -35
>>>   Beginning ww (die) mutex selftests
>>>   All ww mutex selftests passed
>>> 
>>> Besides being a contradiction by terms, the failure to return error
>>> from said worker functions leads to false-positive results.
>>> 
>>> We fix that by adding a *result* variable to *struct stress*, update
>>> it in said worker functions, and finally, in stress(), scan the
>>> *stress_array* and report the first error if any.
>>> 
>>> Fixes: 2a0c11282881 ("locking/ww_mutex: Add kselftests for ww_mutex stress")
>>> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>> 
>> Acked-by: John Stultz <jstultz@google.com>
> 
> I did catch one issue with this patch in testing.  Where as without
> this change, if there was a failure in the first run during
> test_ww_mutex_init(), you could still run the test again manually via
> the sysfs interface without unloading and reloading.
> 
> But with this change, since the error is reported, the module_init
> function sees the error and bails out, removing the sysfs interface.
> If its a module, you can just reload it, but if it is built-in then
> its stuck and can't be rerun.
> 
> So we might want to tweak test_ww_mutex_init() with this patch and
> include something like (whitespace corrupted, just for illustration
> here):
>        ret = run_test_classes();
>        mutex_unlock(&run_lock);
> 
> -       if (!ret)
> +       if (!ret || IS_BUILTIN(CONFIG_WW_MUTEX_SELFTEST))
>                return 0;
> 
>        sysfs_remove_group(test_ww_mutex_kobj, &attr_group);
> 
> ?

Good catch. I must admit I did not test with it built-in. Let me add the above in the fortcoming v2 :-)


Thxs, Håkon

[PATCH 3/4] test-ww_mutex: Handle transient -EDEADLK in test_cycle_work

[Håkon Bugge]

There is a timing issue in test_cycle_work(), in the sense that
acquiring *a_mutex* after deadlock has been detected on the *b_mutex*,
may not succeed immediately. This may lead to false negatives, which
shows up in the log as:

   cyclic deadlock not resolved, ret[77/93] = -35

We fix that by re-trying until the lock is acquired.

Fixes: d1b42b800e5d ("locking/ww_mutex: Add kselftests for resolving ww_mutex cyclic deadlocks")
Fixes: e4a02ed2aaf4 ("locking/ww_mutex: Fix runtime warning in the WW mutex selftest")
Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
---
 kernel/locking/test-ww_mutex.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
index 5a4c92801bdfb..6b29a7a8f5fba 100644
--- a/kernel/locking/test-ww_mutex.c
+++ b/kernel/locking/test-ww_mutex.c
@@ -306,7 +306,9 @@ static void test_cycle_work(struct work_struct *work)
 		err = 0;
 		ww_mutex_unlock(&cycle->a_mutex);
 		ww_mutex_lock_slow(cycle->b_mutex, &ctx);
-		erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
+		do {
+			erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
+		} while (erra == -EDEADLK);
 	}
 
 	if (!err)
-- 
2.43.5

Re: [PATCH 3/4] test-ww_mutex: Handle transient -EDEADLK in test_cycle_work

[John Stultz]

On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>
> There is a timing issue in test_cycle_work(), in the sense that
> acquiring *a_mutex* after deadlock has been detected on the *b_mutex*,
> may not succeed immediately. This may lead to false negatives, which
> shows up in the log as:
>
>    cyclic deadlock not resolved, ret[77/93] = -35
>
> We fix that by re-trying until the lock is acquired.
>

I definitely have seen this error in testing previously.

But is this fix right?  When getting an EDEADLK I thought the ww_mutex
protocol requires the task drop all its locks and re-try acquiring
them all again.


> Fixes: d1b42b800e5d ("locking/ww_mutex: Add kselftests for resolving ww_mutex cyclic deadlocks")
> Fixes: e4a02ed2aaf4 ("locking/ww_mutex: Fix runtime warning in the WW mutex selftest")
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> ---
>  kernel/locking/test-ww_mutex.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
> index 5a4c92801bdfb..6b29a7a8f5fba 100644
> --- a/kernel/locking/test-ww_mutex.c
> +++ b/kernel/locking/test-ww_mutex.c
> @@ -306,7 +306,9 @@ static void test_cycle_work(struct work_struct *work)
>                 err = 0;
>                 ww_mutex_unlock(&cycle->a_mutex);
>                 ww_mutex_lock_slow(cycle->b_mutex, &ctx);
> -               erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
> +               do {
> +                       erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
> +               } while (erra == -EDEADLK);
>         }

I don't have a clear example in mind, but just trying to grab the same
lock again (especially in a loop with no timeout) seems like it could
open up other problems here.

thanks
-john

Re: [PATCH 3/4] test-ww_mutex: Handle transient -EDEADLK in test_cycle_work

[Haakon Bugge]

Hi John,

> On 18 Jun 2026, at 21:04, John Stultz <jstultz@google.com> wrote:
> 
>> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>> 
>> 
>> There is a timing issue in test_cycle_work(), in the sense that
>> acquiring *a_mutex* after deadlock has been detected on the *b_mutex*,
>> may not succeed immediately. This may lead to false negatives, which
>> shows up in the log as:
>> 
>>   cyclic deadlock not resolved, ret[77/93] = -35
>> 
>> We fix that by re-trying until the lock is acquired.
>> 
> 
> I definitely have seen this error in testing previously.
> 
> But is this fix right?  When getting an EDEADLK I thought the ww_mutex
> protocol requires the task drop all its locks and re-try acquiring
> them all again.

This is true for deadlocks. But my main point is that this is *not* a
deadlock which requires one of the WW resolution methods. Put it
another way, if it *is* a deadlock, my do-while loop would loop
forever. It doesn't, and therefore, it isn't a deadlock.

Another question is of course why ww_mutex_lock() returns -EDEADLK
when it isn't the case.

>> Fixes: d1b42b800e5d ("locking/ww_mutex: Add kselftests for resolving ww_mutex cyclic deadlocks")
>> Fixes: e4a02ed2aaf4 ("locking/ww_mutex: Fix runtime warning in the WW mutex selftest")
>> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>> ---
>> kernel/locking/test-ww_mutex.c | 4 +++-
>> 1 file changed, 3 insertions(+), 1 deletion(-)
>> 
>> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
>> index 5a4c92801bdfb..6b29a7a8f5fba 100644
>> --- a/kernel/locking/test-ww_mutex.c
>> +++ b/kernel/locking/test-ww_mutex.c
>> @@ -306,7 +306,9 @@ static void test_cycle_work(struct work_struct *work)
>>                err = 0;
>>                ww_mutex_unlock(&cycle->a_mutex);
>>                ww_mutex_lock_slow(cycle->b_mutex, &ctx);
>> -               erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
>> +               do {
>> +                       erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
>> +               } while (erra == -EDEADLK);
>>        }
>> 
> I don't have a clear example in mind, but just trying to grab the same
> lock again (especially in a loop with no timeout) seems like it could
> open up other problems here.

The do { ... } while (erra == -EDEADLK) loop in test_cycle_work() is
terminating under the ww-mutex progress guarantee. The test builds a
finite circular dependency where worker N waits on worker N+1’s
a_mutex. On -EDEADLK, the worker releases its own a_mutex, acquires
b_mutex through the slow path, and retries. Thus each deadlock
encounter removes one hold from the cycle before retrying. Since the
cycle is finite and every participant follows the same backoff rule,
the deadlock chain must eventually collapse; once the conflicting
owner releases its lock, the retry of a_mutex returns success and the
loop exits.


Thxs, Håkon

Re: [PATCH 3/4] test-ww_mutex: Handle transient -EDEADLK in test_cycle_work

[John Stultz]

On Mon, Jun 22, 2026 at 6:22 AM Haakon Bugge <haakon.bugge@oracle.com> wrote:
> > On 18 Jun 2026, at 21:04, John Stultz <jstultz@google.com> wrote:
> >
> >> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
> >>
> >>
> >> There is a timing issue in test_cycle_work(), in the sense that
> >> acquiring *a_mutex* after deadlock has been detected on the *b_mutex*,
> >> may not succeed immediately. This may lead to false negatives, which
> >> shows up in the log as:
> >>
> >>   cyclic deadlock not resolved, ret[77/93] = -35
> >>
> >> We fix that by re-trying until the lock is acquired.
> >>
> >
> > I definitely have seen this error in testing previously.
> >
> > But is this fix right?  When getting an EDEADLK I thought the ww_mutex
> > protocol requires the task drop all its locks and re-try acquiring
> > them all again.
>
> This is true for deadlocks. But my main point is that this is *not* a
> deadlock which requires one of the WW resolution methods. Put it
> another way, if it *is* a deadlock, my do-while loop would loop
> forever. It doesn't, and therefore, it isn't a deadlock.
>
> Another question is of course why ww_mutex_lock() returns -EDEADLK
> when it isn't the case.

That seems like a solid quetion.

> >> Fixes: d1b42b800e5d ("locking/ww_mutex: Add kselftests for resolving ww_mutex cyclic deadlocks")
> >> Fixes: e4a02ed2aaf4 ("locking/ww_mutex: Fix runtime warning in the WW mutex selftest")
> >> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> >> ---
> >> kernel/locking/test-ww_mutex.c | 4 +++-
> >> 1 file changed, 3 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
> >> index 5a4c92801bdfb..6b29a7a8f5fba 100644
> >> --- a/kernel/locking/test-ww_mutex.c
> >> +++ b/kernel/locking/test-ww_mutex.c
> >> @@ -306,7 +306,9 @@ static void test_cycle_work(struct work_struct *work)
> >>                err = 0;
> >>                ww_mutex_unlock(&cycle->a_mutex);
> >>                ww_mutex_lock_slow(cycle->b_mutex, &ctx);
> >> -               erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
> >> +               do {
> >> +                       erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
> >> +               } while (erra == -EDEADLK);
> >>        }
> >>
> > I don't have a clear example in mind, but just trying to grab the same
> > lock again (especially in a loop with no timeout) seems like it could
> > open up other problems here.
>
> The do { ... } while (erra == -EDEADLK) loop in test_cycle_work() is
> terminating under the ww-mutex progress guarantee. The test builds a
> finite circular dependency where worker N waits on worker N+1’s
> a_mutex. On -EDEADLK, the worker releases its own a_mutex, acquires
> b_mutex through the slow path, and retries. Thus each deadlock
> encounter removes one hold from the cycle before retrying. Since the
> cycle is finite and every participant follows the same backoff rule,
> the deadlock chain must eventually collapse; once the conflicting
> owner releases its lock, the retry of a_mutex returns success and the
> loop exits.
>

Apologies for leaving you hanging on this one for so long! I've
unfortuantely not managed to find the time/focus to really sit and
understand your argument. :(

That said, I have been running testing with your patches applied, and
while they do avoid the "cyclic deadlock not resolved, ret[77/93] =
-35" warnings, I have unfortunately also seen:

[   75.360933] ------------[ cut here ]------------
[   75.361860] DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock)
[   75.361860] WARNING: CPU: 21 PID: 1045 at
kernel/locking/ww_mutex.h:444
__ww_mutex_lock.constprop.0+0x14ca/0x14d0
[   75.372591] CPU: 21 UID: 0 PID: 1045 Comm: kworker/u256:11 Not
tainted 6.18.0-00097-g5d455e26497a-dirty #21 PREEMPT(full)
[   75.379208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 1.17.0-debian-1.17.0-1 04/01/2014
[   75.383205] Workqueue: test-ww_mutex test_cycle_work
[   75.386624] RIP: 0010:__ww_mutex_lock.constprop.0+0x14ca/0x14d0
[   75.390165] Code: e8 5b 05 41 ff 85 c0 74 09 83 3d 10 ec 1a 01 00
74 03 90 eb d3 90 48 c7 c6 30 b2 c9 82 48 c7 c7 0b 36 c9 82 e8 47 e7
d8 fe 0
[   75.401223] RSP: 0000:ffffc90002c57c60 EFLAGS: 00010092
[   75.404873] RAX: 0000000000000000 RBX: ffff888106c6c250 RCX: 0000000000000004
[   75.409870] RDX: 0000000000000001 RSI: ffffc90002c57b08 RDI: 00000000ffffffff
[   75.413929] RBP: ffffc90002c57d30 R08: 00000000ffffbfff R09: ffff8881b8dfffa8
[   75.417866] R10: 40000000ffffc2a2 R11: 0000000000000001 R12: ffff8881032a8000
[   75.423377] R13: ffff888106c6c250 R14: ffffc90002a27cd0 R15: ffffc90002c57d70
[   75.429016] FS:  0000000000000000(0000) GS:ffff888235225000(0000)
knlGS:0000000000000000
[   75.433877] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   75.433877] CR2: 0000000000000000 CR3: 000000000344c001 CR4: 0000000000370ef0
[   75.433877] Call Trace:
[   75.442881]  <TASK>
[   75.442881]  ? test_cycle_work+0x274/0x330
[   75.442881]  ? match_held_lock+0xa5/0xe0
[   75.447874]  ? ww_mutex_lock+0x3c/0xb0
[   75.447874]  ww_mutex_lock+0x3c/0xb0
[   75.447874]  test_cycle_work+0x274/0x330
[   75.454875]  process_one_work+0x201/0x590
[   75.457872]  ? lock_is_held_type+0xcd/0x130
[   75.457872]  worker_thread+0x192/0x350
[   75.457872]  ? __pfx_worker_thread+0x10/0x10
[   75.464893]  kthread+0x11c/0x240
[   75.464929]  ? __pfx_kthread+0x10/0x10
[   75.466859]  ret_from_fork+0x191/0x1e0
[   75.466859]  ? __pfx_kthread+0x10/0x10
[   75.466859]  ret_from_fork_asm+0x1a/0x30
[   75.473863]  </TASK>


I've isolated this down to where this patch adds the  "} while (erra
== -EDEADLK);" loop

[  667.892200] kworker/-961       4d..2. 600168128us :
__ww_mutex_lock.constprop.0: JDB: ww_mutex_lock_acquired ww_ctx:
0000000032d33208 clearing contending_lock 00000000f00df359 to NULL
[  667.898234] kworker/-961       4d..2. 600168129us :
__ww_mutex_lock.constprop.0: JDB: __ww_mutex_kill ww_ctx:
0000000032d33208 setting contending_lock to 000000007c555268
[  667.903876] kworker/-961       4..... 600168130us :
test_cycle_work: JDB: test_cycle_work ww_mutex_lock hit EDEADLK in
loop! ctx: 0000000032d33208
[  667.908993] kworker/-961       4d..2. 600168130us :
__ww_mutex_lock.constprop.0: JDB: __ww_mutex_kill ERRRR ww_ctx:
0000000032d33208 is not NULL! 000000007c555268  trying to set it to
000000007c555268

So while I've not fully gotten my head around the correctness of your
argument (you indeed may be right), it seems your change runs afoul of
the DEBUG_WW_MUTEXES checks, so something extra is needed.

thanks
-john

Re: [PATCH 3/4] test-ww_mutex: Handle transient -EDEADLK in test_cycle_work

[Haakon Bugge]

> On 8 Jul 2026, at 21:55, John Stultz <jstultz@google.com> wrote:
> 
> On Mon, Jun 22, 2026 at 6:22 AM Haakon Bugge <haakon.bugge@oracle.com> wrote:

[snip]

>> 
>> This is true for deadlocks. But my main point is that this is *not* a
>> deadlock which requires one of the WW resolution methods. Put it
>> another way, if it *is* a deadlock, my do-while loop would loop
>> forever. It doesn't, and therefore, it isn't a deadlock.
>> 
>> Another question is of course why ww_mutex_lock() returns -EDEADLK
>> when it isn't the case.
> 
> That seems like a solid quetion.

I think the word is _allegedly_ deadlocked.

>>>> Fixes: d1b42b800e5d ("locking/ww_mutex: Add kselftests for resolving ww_mutex cyclic deadlocks")
>>>> Fixes: e4a02ed2aaf4 ("locking/ww_mutex: Fix runtime warning in the WW mutex selftest")
>>>> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>>>> ---
>>>> kernel/locking/test-ww_mutex.c | 4 +++-
>>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>>> 
>>>> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
>>>> index 5a4c92801bdfb..6b29a7a8f5fba 100644
>>>> --- a/kernel/locking/test-ww_mutex.c
>>>> +++ b/kernel/locking/test-ww_mutex.c
>>>> @@ -306,7 +306,9 @@ static void test_cycle_work(struct work_struct *work)
>>>>               err = 0;
>>>>               ww_mutex_unlock(&cycle->a_mutex);
>>>>               ww_mutex_lock_slow(cycle->b_mutex, &ctx);
>>>> -               erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
>>>> +               do {
>>>> +                       erra = ww_mutex_lock(&cycle->a_mutex, &ctx);
>>>> +               } while (erra == -EDEADLK);
>>>>       }
>>>> 
>>> I don't have a clear example in mind, but just trying to grab the same
>>> lock again (especially in a loop with no timeout) seems like it could
>>> open up other problems here.
>> 
>> The do { ... } while (erra == -EDEADLK) loop in test_cycle_work() is
>> terminating under the ww-mutex progress guarantee. The test builds a
>> finite circular dependency where worker N waits on worker N+1’s
>> a_mutex. On -EDEADLK, the worker releases its own a_mutex, acquires
>> b_mutex through the slow path, and retries. Thus each deadlock
>> encounter removes one hold from the cycle before retrying. Since the
>> cycle is finite and every participant follows the same backoff rule,
>> the deadlock chain must eventually collapse; once the conflicting
>> owner releases its lock, the retry of a_mutex returns success and the
>> loop exits.
>> 
> 
> Apologies for leaving you hanging on this one for so long! I've
> unfortuantely not managed to find the time/focus to really sit and
> understand your argument. :(
> 
> That said, I have been running testing with your patches applied, and
> while they do avoid the "cyclic deadlock not resolved, ret[77/93] =
> -35" warnings, I have unfortunately also seen:
> 
> [   75.360933] ------------[ cut here ]------------
> [   75.361860] DEBUG_LOCKS_WARN_ON(ww_ctx->contending_lock)
> [   75.361860] WARNING: CPU: 21 PID: 1045 at
> kernel/locking/ww_mutex.h:444
> __ww_mutex_lock.constprop.0+0x14ca/0x14d0
> [   75.372591] CPU: 21 UID: 0 PID: 1045 Comm: kworker/u256:11 Not
> tainted 6.18.0-00097-g5d455e26497a-dirty #21 PREEMPT(full)
> [   75.379208] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
> BIOS 1.17.0-debian-1.17.0-1 04/01/2014
> [   75.383205] Workqueue: test-ww_mutex test_cycle_work
> [   75.386624] RIP: 0010:__ww_mutex_lock.constprop.0+0x14ca/0x14d0
> [   75.390165] Code: e8 5b 05 41 ff 85 c0 74 09 83 3d 10 ec 1a 01 00
> 74 03 90 eb d3 90 48 c7 c6 30 b2 c9 82 48 c7 c7 0b 36 c9 82 e8 47 e7
> d8 fe 0
> [   75.401223] RSP: 0000:ffffc90002c57c60 EFLAGS: 00010092
> [   75.404873] RAX: 0000000000000000 RBX: ffff888106c6c250 RCX: 0000000000000004
> [   75.409870] RDX: 0000000000000001 RSI: ffffc90002c57b08 RDI: 00000000ffffffff
> [   75.413929] RBP: ffffc90002c57d30 R08: 00000000ffffbfff R09: ffff8881b8dfffa8
> [   75.417866] R10: 40000000ffffc2a2 R11: 0000000000000001 R12: ffff8881032a8000
> [   75.423377] R13: ffff888106c6c250 R14: ffffc90002a27cd0 R15: ffffc90002c57d70
> [   75.429016] FS:  0000000000000000(0000) GS:ffff888235225000(0000)
> knlGS:0000000000000000
> [   75.433877] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   75.433877] CR2: 0000000000000000 CR3: 000000000344c001 CR4: 0000000000370ef0
> [   75.433877] Call Trace:
> [   75.442881]  <TASK>
> [   75.442881]  ? test_cycle_work+0x274/0x330
> [   75.442881]  ? match_held_lock+0xa5/0xe0
> [   75.447874]  ? ww_mutex_lock+0x3c/0xb0
> [   75.447874]  ww_mutex_lock+0x3c/0xb0
> [   75.447874]  test_cycle_work+0x274/0x330
> [   75.454875]  process_one_work+0x201/0x590
> [   75.457872]  ? lock_is_held_type+0xcd/0x130
> [   75.457872]  worker_thread+0x192/0x350
> [   75.457872]  ? __pfx_worker_thread+0x10/0x10
> [   75.464893]  kthread+0x11c/0x240
> [   75.464929]  ? __pfx_kthread+0x10/0x10
> [   75.466859]  ret_from_fork+0x191/0x1e0
> [   75.466859]  ? __pfx_kthread+0x10/0x10
> [   75.466859]  ret_from_fork_asm+0x1a/0x30
> [   75.473863]  </TASK>
> 
> 
> I've isolated this down to where this patch adds the  "} while (erra
> == -EDEADLK);" loop
> 
> [  667.892200] kworker/-961       4d..2. 600168128us :
> __ww_mutex_lock.constprop.0: JDB: ww_mutex_lock_acquired ww_ctx:
> 0000000032d33208 clearing contending_lock 00000000f00df359 to NULL
> [  667.898234] kworker/-961       4d..2. 600168129us :
> __ww_mutex_lock.constprop.0: JDB: __ww_mutex_kill ww_ctx:
> 0000000032d33208 setting contending_lock to 000000007c555268
> [  667.903876] kworker/-961       4..... 600168130us :
> test_cycle_work: JDB: test_cycle_work ww_mutex_lock hit EDEADLK in
> loop! ctx: 0000000032d33208
> [  667.908993] kworker/-961       4d..2. 600168130us :
> __ww_mutex_lock.constprop.0: JDB: __ww_mutex_kill ERRRR ww_ctx:
> 0000000032d33208 is not NULL! 000000007c555268  trying to set it to
> 000000007c555268
> 
> So while I've not fully gotten my head around the correctness of your
> argument (you indeed may be right), it seems your change runs afoul of
> the DEBUG_WW_MUTEXES checks, so something extra is needed.

Yes, I also see that when running with CONFIG_DEBUG_MUTEXES (using v7.2-rc2). I think this is an allegedly, but not true deadlock. As such, the DEBUG_WW_MUTEXES check does what it is supposed to do.

Instead of retrying the lock acquisition endlessly, I have a recursive helper function in mind, which will be a good citizen and obey the WW rules. Aka, if mutex X is deadlocked, release it, acquire Y using the slow path, retry the acquisition of X, and if deadlocked, call the helper function with X and Y swapped.

This may take a couple of days.


Thxs, Håkon

[PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[Håkon Bugge]

stress_inorder_work() terminates when its timeout expires. If the
final lock acquisition attempt returns -EDEADLK at that point, the
test may report a false failure even though the deadlock would have
been resolved shortly thereafter. Retry a limited number of times
after timeout before reporting failure.

Without this commit, we may see in the log:

   Beginning ww (wound) mutex selftests
   stress (stress_inorder_work) failed with -35

Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
---
 kernel/locking/test-ww_mutex.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
index 6b29a7a8f5fba..95de03179eac0 100644
--- a/kernel/locking/test-ww_mutex.c
+++ b/kernel/locking/test-ww_mutex.c
@@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
 	struct ww_acquire_ctx ctx;
 	int *order;
 	int err;
+	int attempts_after_tmout = 10000;
 
 	stress->result = -ENOMEM;
 
@@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
 			ww_mutex_unlock(&locks[order[n]]);
 
 		if (err == -EDEADLK) {
-			if (!time_after(jiffies, stress->timeout)) {
+			if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
 				ww_mutex_lock_slow(&locks[order[contended]], &ctx);
 				goto retry;
 			}
-- 
2.43.5

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[John Stultz]

On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>
> stress_inorder_work() terminates when its timeout expires. If the
> final lock acquisition attempt returns -EDEADLK at that point, the
> test may report a false failure even though the deadlock would have
> been resolved shortly thereafter. Retry a limited number of times
> after timeout before reporting failure.
>
> Without this commit, we may see in the log:
>
>    Beginning ww (wound) mutex selftests
>    stress (stress_inorder_work) failed with -35
>
> Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> ---
>  kernel/locking/test-ww_mutex.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
> index 6b29a7a8f5fba..95de03179eac0 100644
> --- a/kernel/locking/test-ww_mutex.c
> +++ b/kernel/locking/test-ww_mutex.c
> @@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
>         struct ww_acquire_ctx ctx;
>         int *order;
>         int err;
> +       int attempts_after_tmout = 10000;
>
>         stress->result = -ENOMEM;
>
> @@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
>                         ww_mutex_unlock(&locks[order[n]]);
>
>                 if (err == -EDEADLK) {
> -                       if (!time_after(jiffies, stress->timeout)) {
> +                       if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
>                                 ww_mutex_lock_slow(&locks[order[contended]], &ctx);
>                                 goto retry;
>                         }

Thanks for sending this out! I definitely have seen timeouts hit in my
previous testing. Though, the approach here of "trying harder after
the timeout" at first glance strikes me as like a procrastinator's
panic. :)

It might be nice to have a bit more context in the commit message as
to the soundness of this approach.  If the deadlock resolution takes
longer then we expect in time, is there a reason the number of
iterations tried is also important? Is the key to this change just
maybe ensuring we got enough cputime within the timeout?   ww_mutexes
are intended to ensure forward progress,  but the apparent livelocks
seen prior to cfa92b6d5207 maybe should hint something else might be
going on, so while this may eliminate any timeouts hit due to
scheduling delays, just trying more may not resolve all failures here.

So, no objections with the change here, but maybe its worth improving
the commit message a bit more?

thanks
-john

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[Haakon Bugge]

> On 18 Jun 2026, at 20:55, John Stultz <jstultz@google.com> wrote:
On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:

>> stress_inorder_work() terminates when its timeout expires. If the
>> final lock acquisition attempt returns -EDEADLK at that point, the
>> test may report a false failure even though the deadlock would have
>> been resolved shortly thereafter. Retry a limited number of times
>> after timeout before reporting failure.
>> 
>> Without this commit, we may see in the log:
>> 
>>   Beginning ww (wound) mutex selftests
>>   stress (stress_inorder_work) failed with -35
>> 
>> Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
>> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>> ---
>> kernel/locking/test-ww_mutex.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
>> index 6b29a7a8f5fba..95de03179eac0 100644
>> --- a/kernel/locking/test-ww_mutex.c
>> +++ b/kernel/locking/test-ww_mutex.c
>> @@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
>>        struct ww_acquire_ctx ctx;
>>        int *order;
>>        int err;
>> +       int attempts_after_tmout = 10000;
>> 
>>        stress->result = -ENOMEM;
>> 
>> @@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
>>                        ww_mutex_unlock(&locks[order[n]]);
>> 
>>                if (err == -EDEADLK) {
>> -                       if (!time_after(jiffies, stress->timeout)) {
>> +                       if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
>>                                ww_mutex_lock_slow(&locks[order[contended]], &ctx);
>>                                goto retry;
>>                        }
> 
> Thanks for sending this out! I definitely have seen timeouts hit in my
> previous testing. Though, the approach here of "trying harder after
> the timeout" at first glance strikes me as like a procrastinator's
> panic. :)

Hehe, I think the undersigned is a good example of a procrastinator ;-)

> It might be nice to have a bit more context in the commit message as
> to the soundness of this approach.

I wrote something about it the cover letter:

<quote>
It is worth mentioning that the last patch retries up to 10,000 lock
acquisitions after timeout. This value was determined empirically: 10
and 100 retries were insufficient on the test systems, while 10,000
provided stable results and avoided false negatives.
</quote>

Shall I add that to the commit message?

> If the deadlock resolution takes
> longer then we expect in time,

That is not the case. When we terminate due to timeout, we have no
resolution of the deadlock. I view it as there is a probability we run
into deadlock, let's say 10%, for each iteration. Now, if we terminate
due to timeout - and - we are unlucky and the last iteration
deadlocked, we cannot just return -EDEADLK, as that will mark the test
as failed. Instead, we retry some more iterations in order to resolve
the deadlock.

> is there a reason the number of
> iterations tried is also important? Is the key to this change just
> maybe ensuring we got enough cputime within the timeout?   ww_mutexes
> are intended to ensure forward progress,  but the apparent livelocks
> seen prior to cfa92b6d5207 maybe should hint something else might be
> going on, so while this may eliminate any timeouts hit due to
> scheduling delays, just trying more may not resolve all failures here.
> 
> So, no objections with the change here, but maybe its worth improving
> the commit message a bit more?

Yes, let me know how you feel about adding what I wrote in the
cover-letter quoted above to the commit message.

Thxs, Håkon

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[Haakon Bugge]

> On 18 Jun 2026, at 20:55, John Stultz <jstultz@google.com> wrote:
> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
> 
> stress_inorder_work() terminates when its timeout expires. If the
> final lock acquisition attempt returns -EDEADLK at that point, the
> test may report a false failure even though the deadlock would have
> been resolved shortly thereafter. Retry a limited number of times
> after timeout before reporting failure.
> 
> Without this commit, we may see in the log:
> 
>  Beginning ww (wound) mutex selftests
>  stress (stress_inorder_work) failed with -35
> 
> Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> ---
> kernel/locking/test-ww_mutex.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
> index 6b29a7a8f5fba..95de03179eac0 100644
> --- a/kernel/locking/test-ww_mutex.c
> +++ b/kernel/locking/test-ww_mutex.c
> @@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
>       struct ww_acquire_ctx ctx;
>       int *order;
>       int err;
> +       int attempts_after_tmout = 10000;
> 
>       stress->result = -ENOMEM;
> 
> @@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
>                       ww_mutex_unlock(&locks[order[n]]);
> 
>               if (err == -EDEADLK) {
> -                       if (!time_after(jiffies, stress->timeout)) {
> +                       if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
>                               ww_mutex_lock_slow(&locks[order[contended]], &ctx);
>                               goto retry;
>                       }
> 
> Thanks for sending this out! I definitely have seen timeouts hit in my
> previous testing. Though, the approach here of "trying harder after
> the timeout" at first glance strikes me as like a procrastinator's
> panic. :)
> 
> Hehe, I think the undersigned is a good example of a procrastinator ;-)
> 
> It might be nice to have a bit more context in the commit message as
> to the soundness of this approach.
> 
> I wrote something about it the cover letter:
> 
> <quote>
> It is worth mentioning that the last patch retries up to 10,000 lock
> acquisitions after timeout. This value was determined empirically: 10
> and 100 retries were insufficient on the test systems, while 10,000
> provided stable results and avoided false negatives.
> </quote>
> 
> Shall I add that to the commit message?
> 
> If the deadlock resolution takes
> longer then we expect in time,
> 
> That is not the case. When we terminate due to timeout, we have no
> resolution of the deadlock. I view it as there is a probability we run
> into deadlock, let's say 10%, for each iteration. Now, if we terminate
> due to timeout - and - we are unlucky and the last iteration
> deadlocked, we cannot just return -EDEADLK, as that will mark the test
> as failed. Instead, we retry some more iterations in order to resolve
> the deadlock.
> 
> is there a reason the number of
> iterations tried is also important? Is the key to this change just
> maybe ensuring we got enough cputime within the timeout?   ww_mutexes
> are intended to ensure forward progress,  but the apparent livelocks
> seen prior to cfa92b6d5207 maybe should hint something else might be
> going on, so while this may eliminate any timeouts hit due to
> scheduling delays, just trying more may not resolve all failures here.
> 
> So, no objections with the change here, but maybe its worth improving
> the commit message a bit more?
> 
> Yes, let me know how you feel about adding what I wrote in the
> cover-letter quoted above to the commit message.
> 
> Thxs, Håkon

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[Haakon Bugge]

> On 6 Jul 2026, at 11:01, Haakon Bugge <haakon.bugge@oracle.com> wrote:
> 
> 
>> On 18 Jun 2026, at 20:55, John Stultz <jstultz@google.com> wrote:
>> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>> 
>> stress_inorder_work() terminates when its timeout expires. If the
>> final lock acquisition attempt returns -EDEADLK at that point, the
>> test may report a false failure even though the deadlock would have
>> been resolved shortly thereafter. Retry a limited number of times
>> after timeout before reporting failure.
>> 
>> Without this commit, we may see in the log:
>> 
>> Beginning ww (wound) mutex selftests
>> stress (stress_inorder_work) failed with -35
>> 
>> Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
>> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
>> ---
>> kernel/locking/test-ww_mutex.c | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>> 
>> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
>> index 6b29a7a8f5fba..95de03179eac0 100644
>> --- a/kernel/locking/test-ww_mutex.c
>> +++ b/kernel/locking/test-ww_mutex.c
>> @@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
>>      struct ww_acquire_ctx ctx;
>>      int *order;
>>      int err;
>> +       int attempts_after_tmout = 10000;
>> 
>>      stress->result = -ENOMEM;
>> 
>> @@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
>>                      ww_mutex_unlock(&locks[order[n]]);
>> 
>>              if (err == -EDEADLK) {
>> -                       if (!time_after(jiffies, stress->timeout)) {
>> +                       if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
>>                              ww_mutex_lock_slow(&locks[order[contended]], &ctx);
>>                              goto retry;
>>                      }
>> 
>> Thanks for sending this out! I definitely have seen timeouts hit in my
>> previous testing. Though, the approach here of "trying harder after
>> the timeout" at first glance strikes me as like a procrastinator's
>> panic. :)
>> 
>> Hehe, I think the undersigned is a good example of a procrastinator ;-)
>> 
>> It might be nice to have a bit more context in the commit message as
>> to the soundness of this approach.
>> 
>> I wrote something about it the cover letter:
>> 
>> <quote>
>> It is worth mentioning that the last patch retries up to 10,000 lock
>> acquisitions after timeout. This value was determined empirically: 10
>> and 100 retries were insufficient on the test systems, while 10,000
>> provided stable results and avoided false negatives.
>> </quote>
>> 
>> Shall I add that to the commit message?
>> 
>> If the deadlock resolution takes
>> longer then we expect in time,
>> 
>> That is not the case. When we terminate due to timeout, we have no
>> resolution of the deadlock. I view it as there is a probability we run
>> into deadlock, let's say 10%, for each iteration. Now, if we terminate
>> due to timeout - and - we are unlucky and the last iteration
>> deadlocked, we cannot just return -EDEADLK, as that will mark the test
>> as failed. Instead, we retry some more iterations in order to resolve
>> the deadlock.
>> 
>> is there a reason the number of
>> iterations tried is also important? Is the key to this change just
>> maybe ensuring we got enough cputime within the timeout?   ww_mutexes
>> are intended to ensure forward progress,  but the apparent livelocks
>> seen prior to cfa92b6d5207 maybe should hint something else might be
>> going on, so while this may eliminate any timeouts hit due to
>> scheduling delays, just trying more may not resolve all failures here.
>> 
>> So, no objections with the change here, but maybe its worth improving
>> the commit message a bit more?
>> 
>> Yes, let me know how you feel about adding what I wrote in the
>> cover-letter quoted above to the commit message.


A gentle ping on this series!


Thxs, Håkon

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[John Stultz]

On Mon, Jun 22, 2026 at 6:48 AM Haakon Bugge <haakon.bugge@oracle.com> wrote:
> > On 18 Jun 2026, at 20:55, John Stultz <jstultz@google.com> wrote:
> On Wed, Jun 17, 2026 at 5:13 AM Håkon Bugge <haakon.bugge@oracle.com> wrote:
>
> >> stress_inorder_work() terminates when its timeout expires. If the
> >> final lock acquisition attempt returns -EDEADLK at that point, the
> >> test may report a false failure even though the deadlock would have
> >> been resolved shortly thereafter. Retry a limited number of times
> >> after timeout before reporting failure.
> >>
> >> Without this commit, we may see in the log:
> >>
> >>   Beginning ww (wound) mutex selftests
> >>   stress (stress_inorder_work) failed with -35
> >>
> >> Fixes: cfa92b6d5207 ("locking/ww_mutex/test: Make sure we bail out instead of livelock")
> >> Signed-off-by: Håkon Bugge <haakon.bugge@oracle.com>
> >> ---
> >> kernel/locking/test-ww_mutex.c | 3 ++-
> >> 1 file changed, 2 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/kernel/locking/test-ww_mutex.c b/kernel/locking/test-ww_mutex.c
> >> index 6b29a7a8f5fba..95de03179eac0 100644
> >> --- a/kernel/locking/test-ww_mutex.c
> >> +++ b/kernel/locking/test-ww_mutex.c
> >> @@ -444,6 +444,7 @@ static void stress_inorder_work(struct work_struct *work)
> >>        struct ww_acquire_ctx ctx;
> >>        int *order;
> >>        int err;
> >> +       int attempts_after_tmout = 10000;
> >>
> >>        stress->result = -ENOMEM;
> >>
> >> @@ -476,7 +477,7 @@ static void stress_inorder_work(struct work_struct *work)
> >>                        ww_mutex_unlock(&locks[order[n]]);
> >>
> >>                if (err == -EDEADLK) {
> >> -                       if (!time_after(jiffies, stress->timeout)) {
> >> +                       if ((!time_after(jiffies, stress->timeout)) || attempts_after_tmout--) {
> >>                                ww_mutex_lock_slow(&locks[order[contended]], &ctx);
> >>                                goto retry;
> >>                        }
> >
> > Thanks for sending this out! I definitely have seen timeouts hit in my
> > previous testing. Though, the approach here of "trying harder after
> > the timeout" at first glance strikes me as like a procrastinator's
> > panic. :)
>
> Hehe, I think the undersigned is a good example of a procrastinator ;-)
>
> > It might be nice to have a bit more context in the commit message as
> > to the soundness of this approach.
>
> I wrote something about it the cover letter:
>
> <quote>
> It is worth mentioning that the last patch retries up to 10,000 lock
> acquisitions after timeout. This value was determined empirically: 10
> and 100 retries were insufficient on the test systems, while 10,000
> provided stable results and avoided false negatives.
> </quote>
>
> Shall I add that to the commit message?

That would be great.

> > If the deadlock resolution takes
> > longer then we expect in time,
>
> That is not the case. When we terminate due to timeout, we have no
> resolution of the deadlock. I view it as there is a probability we run
> into deadlock, let's say 10%, for each iteration. Now, if we terminate
> due to timeout - and - we are unlucky and the last iteration
> deadlocked, we cannot just return -EDEADLK, as that will mark the test
> as failed. Instead, we retry some more iterations in order to resolve
> the deadlock.

But with the retry count needing to be set to 10,000 it seems like
more then just being unlucky on the last iteration, no?

I'm still a bit wary on this change, but I am very eager to see these
occasional failures be sorted out, so thank you for your efforts here!

thanks
-john

Re: [PATCH 4/4] test-ww_mutex: Retry lock acquisition after timeout if deadlocked

[Haakon Bugge]

> On 8 Jul 2026, at 22:01, John Stultz <jstultz@google.com> wrote:
> 
> On Mon, Jun 22, 2026 at 6:48 AM Haakon Bugge <haakon.bugge@oracle.com> wrote:

[snip]

>> <quote>
>> It is worth mentioning that the last patch retries up to 10,000 lock
>> acquisitions after timeout. This value was determined empirically: 10
>> and 100 retries were insufficient on the test systems, while 10,000
>> provided stable results and avoided false negatives.
>> </quote>
>> 
>> Shall I add that to the commit message?
> 
> That would be great.
> 
>>> If the deadlock resolution takes
>>> longer then we expect in time,
>> 
>> That is not the case. When we terminate due to timeout, we have no
>> resolution of the deadlock. I view it as there is a probability we run
>> into deadlock, let's say 10%, for each iteration. Now, if we terminate
>> due to timeout - and - we are unlucky and the last iteration
>> deadlocked, we cannot just return -EDEADLK, as that will mark the test
>> as failed. Instead, we retry some more iterations in order to resolve
>> the deadlock.
> 
> But with the retry count needing to be set to 10,000 it seems like
> more then just being unlucky on the last iteration, no?

Good observation. Very unlucky may be?

> I'm still a bit wary on this change, but I am very eager to see these
> occasional failures be sorted out, so thank you for your efforts here!

Another way, if the last iteration fails, it to, let's say, double the timeout and continue, but return on the first successful iteration. But, it is the same fix in new wrapping.

Let me add some statistics in a v2, to see if that could ease your caution here.


Thxs, Håkon