[PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Ameer Hamza]

When a local request reuses an inactive rsb, find_rsb_dir() sets
RSB_MASTER_UNCERTAIN if res_master_nodeid is not our nodeid. This also
matches an rsb that never got a master (res_master_nodeid 0, res_nodeid
-1). set_master() then trusts the "cached" master, copies res_nodeid -1
into lkb_nodeid and skips the lookup, and _request_lock() hits the
DLM_ASSERT(r->res_nodeid >= 0) in is_remote():

  rsb: nodeid -1 master 0 dir 1 flags 200 first 1 rlc 0 name pr
  Kernel panic - not syncing: DLM:  Record message above and reboot.
  Call Trace:
   _request_lock.cold+0x3e/0x3e [dlm]
   request_lock+0xcc/0x110 [dlm]
   dlm_user_request+0x110/0x340 [dlm]
   device_write+0x605/0x6d0 [dlm]

Such an rsb is left behind when a request is cancelled while its lookup
is still outstanding and the lockspace then recovers, e.g. because the
dir node died: dlm_recover_waiters_post() completes the cancelled lkb
instead of resending it, so no master is ever set. Once the rsb is
inactive, the next request for the name revives it and crashes as
above. If we became the dir node in that recovery, deactivate_rsb()
even keeps the rsb as a dir record, so it does not age out.

Before commit c04fecb4d9f7 ("dlm: use rsbtbl as resource directory")
the flag was only set when res_nodeid was > 0. Restore that and set the
flag only when a master is actually cached, so that set_master() does a
normal lookup (or becomes the master itself as dir node) for an rsb
that has none. res_first_lkid is still cleared, as the cancelled lkb
leaves a stale id behind.

Fixes: c04fecb4d9f7 ("dlm: use rsbtbl as resource directory")
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ameer Hamza <ameer.hamza@truenas.com>
---
Reproduced with two nodes: pause the dir node for a name, request the
name, cancel the request once the lookup is out, remove the paused node,
request the name again. The second request asserts without this patch
and is granted with it, also when the dir node moves to a third live
node.

 fs/dlm/lock.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index c715e171bab7..2323c7709aa1 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -826,8 +826,12 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,
 
 	if (from_local && (r->res_master_nodeid != our_nodeid)) {
 		/* Because we have held no locks on this rsb,
-		   res_master_nodeid could have become stale. */
-		rsb_set_flag(r, RSB_MASTER_UNCERTAIN);
+		 * res_master_nodeid could have become stale.  If no master
+		 * was ever set, there is nothing to confirm and set_master()
+		 * must resolve it as for a new rsb.
+		 */
+		if (r->res_master_nodeid)
+			rsb_set_flag(r, RSB_MASTER_UNCERTAIN);
 		r->res_first_lkid = 0;
 	}
 
-- 
2.53.0

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Ameer Hamza]

On Mon, Oct 19, 2026 at 08:16:27PM +0500, Ameer Hamza wrote:
> When a local request reuses an inactive rsb, find_rsb_dir() sets
> RSB_MASTER_UNCERTAIN if res_master_nodeid is not our nodeid. This also
> matches an rsb that never got a master (res_master_nodeid 0, res_nodeid
> -1).

Gentle ping.

Thanks,
Ameer Hamza

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Alexander Aring]

Hi,

On Mon, Oct 19, 2026 at 11:17 AM Ameer Hamza <ameer.hamza@truenas.com> wrote:
>
> When a local request reuses an inactive rsb, find_rsb_dir() sets
> RSB_MASTER_UNCERTAIN if res_master_nodeid is not our nodeid. This also
> matches an rsb that never got a master (res_master_nodeid 0, res_nodeid
> -1). set_master() then trusts the "cached" master, copies res_nodeid -1
> into lkb_nodeid and skips the lookup, and _request_lock() hits the
> DLM_ASSERT(r->res_nodeid >= 0) in is_remote():
>
>   rsb: nodeid -1 master 0 dir 1 flags 200 first 1 rlc 0 name pr
>   Kernel panic - not syncing: DLM:  Record message above and reboot.
>   Call Trace:
>    _request_lock.cold+0x3e/0x3e [dlm]
>    request_lock+0xcc/0x110 [dlm]
>    dlm_user_request+0x110/0x340 [dlm]
>    device_write+0x605/0x6d0 [dlm]
>
> Such an rsb is left behind when a request is cancelled while its lookup
> is still outstanding and the lockspace then recovers, e.g. because the
> dir node died: dlm_recover_waiters_post() completes the cancelled lkb
> instead of resending it, so no master is ever set. Once the rsb is
> inactive, the next request for the name revives it and crashes as
> above. If we became the dir node in that recovery, deactivate_rsb()
> even keeps the rsb as a dir record, so it does not age out.
>
> Before commit c04fecb4d9f7 ("dlm: use rsbtbl as resource directory")
> the flag was only set when res_nodeid was > 0. Restore that and set the
> flag only when a master is actually cached, so that set_master() does a
> normal lookup (or becomes the master itself as dir node) for an rsb
> that has none. res_first_lkid is still cleared, as the cancelled lkb
> leaves a stale id behind.
>
> Fixes: c04fecb4d9f7 ("dlm: use rsbtbl as resource directory")
> Assisted-by: Claude:claude-opus-5-5
> Signed-off-by: Ameer Hamza <ameer.hamza@truenas.com>
> ---
> Reproduced with two nodes: pause the dir node for a name, request the
> name, cancel the request once the lookup is out, remove the paused node,
> request the name again. The second request asserts without this patch
> and is granted with it, also when the dir node moves to a third live
> node.
>

question:

how do you "pause" anything in DLM?

>  fs/dlm/lock.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
> index c715e171bab7..2323c7709aa1 100644
> --- a/fs/dlm/lock.c
> +++ b/fs/dlm/lock.c
> @@ -826,8 +826,12 @@ static int find_rsb_dir(struct dlm_ls *ls, const void *name, int len,
>
>         if (from_local && (r->res_master_nodeid != our_nodeid)) {
>                 /* Because we have held no locks on this rsb,
> -                  res_master_nodeid could have become stale. */
> -               rsb_set_flag(r, RSB_MASTER_UNCERTAIN);
> +                * res_master_nodeid could have become stale.  If no master
> +                * was ever set, there is nothing to confirm and set_master()
> +                * must resolve it as for a new rsb.
> +                */
> +               if (r->res_master_nodeid)
> +                       rsb_set_flag(r, RSB_MASTER_UNCERTAIN);
>                 r->res_first_lkid = 0;
>         }

I tried this patch and no significant failure occurred, although your
case is difficult to hit with a normal workload. So I will ack this
patch...

Acked-by: Alexander Aring <aahringo@redhat.com>

This reminds me that there are too many places that determine the
master or when the master isn't set, instead of having one central
place. I tried to improve this situation, but the changes seemed too
extensive to be accepted upstream. [0]

- Alex

[0] https://lore.kernel.org/gfs2/20240423195559.1527187-9-aahringo@redhat.com/

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Ameer Hamza]

On Mon, Nov 02, 2026 at 02:49:04PM -0500, Alexander Aring wrote:
> question:
> 
> how do you "pause" anything in DLM?

The nodes are VMs, so I freeze the dir node from the QEMU monitor. It
stays a member but never answers the lookup, and after the cancel I
evict it through configfs. That is what happens on our HA systems when
the peer reboots without leaving the lockspace and a pending request
gets cancelled before the peer is evicted.

> Acked-by: Alexander Aring <aahringo@redhat.com>

Thanks for testing and the ack.

Best Regards,
Ameer Hamza

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Alexander Aring]

Hi,

On Mon, Nov 2, 2026 at 3:03 PM Ameer Hamza <ameer.hamza@truenas.com> wrote:
>
> On Mon, Nov 02, 2026 at 02:49:04PM -0500, Alexander Aring wrote:
> > question:
> >
> > how do you "pause" anything in DLM?
>
> The nodes are VMs, so I freeze the dir node from the QEMU monitor. It
> stays a member but never answers the lookup, and after the cancel I
> evict it through configfs. That is what happens on our HA systems when
> the peer reboots without leaving the lockspace and a pending request
> gets cancelled before the peer is evicted.

Still, do you just randomly freeze it, or do you busy/poll wait on
specific struct field changes?

- Alex

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Ameer Hamza]

On Mon, Nov 02, 2026 at 03:38:12PM -0500, Alexander Aring wrote:
> Hi,
> 
> On Mon, Nov 2, 2026 at 3:03 PM Ameer Hamza <ameer.hamza@truenas.com> wrote:
> >
> > On Mon, Nov 02, 2026 at 02:49:04PM -0500, Alexander Aring wrote:
> > > question:
> > >
> > > how do you "pause" anything in DLM?
> >
> > The nodes are VMs, so I freeze the dir node from the QEMU monitor. It
> > stays a member but never answers the lookup, and after the cancel I
> > evict it through configfs. That is what happens on our HA systems when
> > the peer reboots without leaving the lockspace and a pending request
> > gets cancelled before the peer is evicted.
> 
> Still, do you just randomly freeze it, or do you busy/poll wait on
> specific struct field changes?

I don't poll anything. The peer is frozen before the request is made,
so the lookup to it can never complete. Then the request is cancelled,
the peer is evicted and the name is requested again, and it asserts
every time.

Best Regards,
Ameer Hamza

Re: [PATCH] dlm: don't set RSB_MASTER_UNCERTAIN on rsb with no master

[Alexander Aring]

Hi,

On Mon, Nov 2, 2026 at 4:03 PM Ameer Hamza <ameer.hamza@truenas.com> wrote:
>
> On Mon, Nov 02, 2026 at 03:38:12PM -0500, Alexander Aring wrote:
> > Hi,
> >
> > On Mon, Nov 2, 2026 at 3:03 PM Ameer Hamza <ameer.hamza@truenas.com> wrote:
> > >
> > > On Mon, Nov 02, 2026 at 02:49:04PM -0500, Alexander Aring wrote:
> > > > question:
> > > >
> > > > how do you "pause" anything in DLM?
> > >
> > > The nodes are VMs, so I freeze the dir node from the QEMU monitor. It
> > > stays a member but never answers the lookup, and after the cancel I
> > > evict it through configfs. That is what happens on our HA systems when
> > > the peer reboots without leaving the lockspace and a pending request
> > > gets cancelled before the peer is evicted.
> >
> > Still, do you just randomly freeze it, or do you busy/poll wait on
> > specific struct field changes?
>
> I don't poll anything. The peer is frozen before the request is made,
> so the lookup to it can never complete. Then the request is cancelled,
> the peer is evicted and the name is requested again, and it asserts
> every time.
>

I see, do you use dlm_controld/corosync as cluster manager?

I would assume at some point they should fence the frozen peer.

- Alex