From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D7D122A4EF for ; Mon, 13 Jan 2025 12:07:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736770041; cv=none; b=pVg5GffmSgEByAC7qVTxPKcvz48Up/c9+KHcn4dY7gHYwPZebKeCgvdEPgoS2LA9pbpcy/kkqNamzk1TTtrXp3lZbXS3dE1jsARDo2kCT9GQPXJ0lFGGMYgr7AC82Rab2LyvIPma+VbbgoMGk2fgwITYNW9lLHDgFsooRvwrNJE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736770041; c=relaxed/simple; bh=OsuGXX1IB+4jMhYbVJJTnzxRgsURfdW1BWdB4B7uvHI=; h=Message-ID:Subject:From:To:Cc:Date:In-Reply-To:References: Content-Type:MIME-Version; b=pN3QFKdxBPlIriyp4OGU1WTIUp+XSr1TkEKIclC+kXH0z9OzWTSS2vRF+MxouPzOo4jsrQ+Sb1UpDSva9DIsMofXmGO7vGY8iAsSC6ClWvX3PAMz1wZt/M3ka3wz636hNuwGqB6KizhkHHktYbgTZqLyVW6aQa8sjDbko6KSIIM= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=EOxiqTp9; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="EOxiqTp9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736770038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+TdSHXspwnZXwF4h2DZEwKqf77S3RUPoHliyOCG8joM=; b=EOxiqTp9TK0SDOjXvuGok8AXFMn6c8rV7GwMvW9rGbI4xQeZ1BYmf/7WAsegGxVG0iRzB3 eXvrFX8GUlge4Mo5pZyyzjlcDeE/HTsEMgmhCmtOjXKBpDFlKzPe7x11YlUCpU6fiVXzaR HtCgiDTstk3QI/9esSF4opujsWxRFBQ= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-660-QHgLw381OU2Z5mOJ0EUPuA-1; Mon, 13 Jan 2025 07:07:17 -0500 X-MC-Unique: QHgLw381OU2Z5mOJ0EUPuA-1 X-Mimecast-MFC-AGG-ID: QHgLw381OU2Z5mOJ0EUPuA Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38629a685fdso1499745f8f.2 for ; Mon, 13 Jan 2025 04:07:17 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736770036; x=1737374836; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:cc:to:from:subject:message-id:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+TdSHXspwnZXwF4h2DZEwKqf77S3RUPoHliyOCG8joM=; b=kkSyjMFNb5Ea7iUdgL/K2FL198eF2uR7UTsl4mrq4fjgmRh5csKC7yZY2ZAlCOQUb3 5sdgNQXL8nHmhhxNBX6Ytshv1pnPwg+VRGj5gMEMHQ8AXZ4De24IU8l0k+sIkTAInVab JTAGU2ZDdefCdZebERQ5Gq+dxfCv1vKpWpqz8s9ectra4fz1EH1Aph/UFfuwDMv4O0qE /kjG54Py0Pni5qEZt62Q8esROzOKf1efxxhjupt7sTrQLLgJW1I5FJAABgey/x71G68N ics+nr8cGZiUj2ukxjp2K6qPd+YNXfJtr8Oqpo69TDzogjAhx+m+AsjomXQkSCSMUBOM N6Wg== X-Forwarded-Encrypted: i=1; AJvYcCX2orpEpSyXBmyfy8gqjt5vHAdBo9w1FfmU0JNcOhDs0BPKyB+FnlGlwz+dxUdM4+lLNYD/pqRr3B+A+KE=@vger.kernel.org X-Gm-Message-State: AOJu0YyGbWgi+cNWDnMDxs7npHOtynKOty6Mzvxt+MicIisY9B2/QLgI I3Q1sbHUMFo7MB9Yv0t7mnTvSsljDG1kvi0p3NPiHoB8AjMYcaxDo3VQe4NXK6yKPNPBLvd+3FC JQZ8ztfStZV+MjjfgxOr7ivyNs8CDojE8CGR4sCRYiuCLvNmDOThU/ol2ZpBwnw== X-Gm-Gg: ASbGncvPE6+86j87s9oGLUMlvTAOfoZ3x8SXSLjGwV5/gq0EwlrRLrI6qDHpdjrHpdF gwfQhHEoyuMqNVKk3zSdlwqTBGTRYMjJ2M6GYpiiOIOQby8s6yM+BVYmP83IGqCTL2dvlujU1Az qoaY5rw6HSPmtixvw3pmrDuQ6r9txcLxjmqDjZiC76NLB3NIL+Ga3TU3JEptdYCyAh7Gn4boXLW 9wSDl+7SA+AM4M7mBm75S0VouJI5Y/4scvxR/oj07psqlFGArENR8DCplagGM2RkhUs5nElY/hS UORvsJ1k5y61oHBGe6BbloAUjlksVOKvtrkXM4eTgJfbx23zwxYmeSfu1tUKOtCZZC+z+RV91Ho 2qmps2HwhNFy+RY1J X-Received: by 2002:a5d:64af:0:b0:38a:8e2e:9fcc with SMTP id ffacd0b85a97d-38a8e2ea11cmr15492270f8f.45.1736770035852; Mon, 13 Jan 2025 04:07:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IGpSJ9ro6ZHAQTx4w8s1hVHKl20+St+yEc7qTxW7zB/9BUgzdP9J0EacCpHEFeFW46e29u8xQ== X-Received: by 2002:a5d:64af:0:b0:38a:8e2e:9fcc with SMTP id ffacd0b85a97d-38a8e2ea11cmr15492222f8f.45.1736770035376; Mon, 13 Jan 2025 04:07:15 -0800 (PST) Received: from ?IPv6:2001:16b8:2d84:4500:c248:649d:c975:50e7? (200116b82d844500c248649dc97550e7.dip.versatel-1u1.de. [2001:16b8:2d84:4500:c248:649d:c975:50e7]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38a8e3834a6sm11892576f8f.28.2025.01.13.04.07.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 13 Jan 2025 04:07:15 -0800 (PST) Message-ID: <24a1169b0d1a4ba895df30c2665265bbf3e78e2a.camel@redhat.com> Subject: Re: [PATCH 1/3] drm/sched: Document run_job() refcount hazard From: Philipp Stanner To: Christian =?ISO-8859-1?Q?K=F6nig?= , Philipp Stanner , Luben Tuikov , Matthew Brost , Danilo Krummrich , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Simona Vetter , Sumit Semwal Cc: dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org Date: Mon, 13 Jan 2025 13:07:13 +0100 In-Reply-To: References: <20250109133710.39404-2-phasta@kernel.org> <20250109133710.39404-4-phasta@kernel.org> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.52.4 (3.52.4-2.fc40) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 On Thu, 2025-01-09 at 15:01 +0100, Christian K=C3=B6nig wrote: > Am 09.01.25 um 14:37 schrieb Philipp Stanner: > > From: Philipp Stanner > >=20 > > drm_sched_backend_ops.run_job() returns a dma_fence for the > > scheduler. > > That fence is signalled by the driver once the hardware completed > > the > > associated job. The scheduler does not increment the reference > > count on > > that fence, but implicitly expects to inherit this fence from > > run_job(). > >=20 > > This is relatively subtle and prone to misunderstandings. > >=20 > > This implies that, to keep a reference for itself, a driver needs > > to > > call dma_fence_get() in addition to dma_fence_init() in that > > callback. > >=20 > > It's further complicated by the fact that the scheduler even > > decrements > > the refcount in drm_sched_run_job_work() since it created a new > > reference in drm_sched_fence_scheduled(). It does, however, still > > use > > its pointer to the fence after calling dma_fence_put() - which is > > safe > > because of the aforementioned new reference, but actually still > > violates > > the refcounting rules. > >=20 > > Improve the explanatory comment for that decrement. > >=20 > > Move the call to dma_fence_put() to the position behind the last > > usage > > of the fence. > >=20 > > Document the necessity to increment the reference count in > > drm_sched_backend_ops.run_job(). > >=20 > > Signed-off-by: Philipp Stanner > > --- > > =C2=A0 drivers/gpu/drm/scheduler/sched_main.c | 10 +++++++--- > > =C2=A0 include/drm/gpu_scheduler.h=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 19 +++++++++++++++---- > > =C2=A0 2 files changed, 22 insertions(+), 7 deletions(-) > >=20 > > diff --git a/drivers/gpu/drm/scheduler/sched_main.c > > b/drivers/gpu/drm/scheduler/sched_main.c > > index 57da84908752..5f46c01eb01e 100644 > > --- a/drivers/gpu/drm/scheduler/sched_main.c > > +++ b/drivers/gpu/drm/scheduler/sched_main.c > > @@ -1218,15 +1218,19 @@ static void drm_sched_run_job_work(struct > > work_struct *w) > > =C2=A0=C2=A0 drm_sched_fence_scheduled(s_fence, fence); > > =C2=A0=20 > > =C2=A0=C2=A0 if (!IS_ERR_OR_NULL(fence)) { > > - /* Drop for original kref_init of the fence */ > > - dma_fence_put(fence); > > - > > =C2=A0=C2=A0 r =3D dma_fence_add_callback(fence, &sched_job->cb, > > =C2=A0=C2=A0 =C2=A0=C2=A0 drm_sched_job_done_cb); > > =C2=A0=C2=A0 if (r =3D=3D -ENOENT) > > =C2=A0=C2=A0 drm_sched_job_done(sched_job, fence- > > >error); > > =C2=A0=C2=A0 else if (r) > > =C2=A0=C2=A0 DRM_DEV_ERROR(sched->dev, "fence add > > callback failed (%d)\n", r); > > + > > + /* > > + * s_fence took a new reference to fence in the > > call to > > + * drm_sched_fence_scheduled() above. The > > reference passed by > > + * run_job() above is now not needed any longer. > > Drop it. > > + */ > > + dma_fence_put(fence); > > =C2=A0=C2=A0 } else { > > =C2=A0=C2=A0 drm_sched_job_done(sched_job, IS_ERR(fence) ? > > =C2=A0=C2=A0 =C2=A0=C2=A0 PTR_ERR(fence) : 0); > > diff --git a/include/drm/gpu_scheduler.h > > b/include/drm/gpu_scheduler.h > > index 95e17504e46a..d5cd2a78f27c 100644 > > --- a/include/drm/gpu_scheduler.h > > +++ b/include/drm/gpu_scheduler.h > > @@ -420,10 +420,21 @@ struct drm_sched_backend_ops { > > =C2=A0=C2=A0 struct drm_sched_entity > > *s_entity); > > =C2=A0=20 > > =C2=A0=C2=A0 /** > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * @run_job: Called to= execute the job once all of the > > dependencies > > -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * have been resolved.= =C2=A0 This may be called multiple times, > > if > > - * timedout_job() has happened and > > drm_sched_job_recovery() > > - * decides to try it again. > > + * @run_job: Called to execute the job once all of the > > dependencies > > + * have been resolved. This may be called multiple times, > > if > > + * timedout_job() has happened and > > drm_sched_job_recovery() decides to > > + * try it again. >=20 > I just came to realize that this hack with calling run_job multiple=20 > times won't work any more with this patch here. >=20 > The background is that you can't allocate memory for a newly returned > fence and as far as I know no driver pre allocates multiple HW fences > for a job. >=20 > So at least amdgpu used to re-use the same HW fence as return over > and=20 > over again, just re-initialized the reference count. I removed that > hack=20 > from amdgpu, but just FYI it could be that other driver did the same. >=20 > Apart from that concern I think that this patch is really the right=20 > thing and that driver hacks relying on the order of dropping > references=20 > are fundamentally broken approaches. I don't see how a hack couldn't work anymore with this patch. All it does is add comments, and it moves the putting of the dma_fence to where it belongs. But we're in run_job_work() here, which executes sequentially. Drivers reusing fences or modifying the refcount might be bad and hacky, but as long as we're not racing here it doesn't matter whether the scheduler decrements the refcount a few nanoseconds later than before. Anyways, even if I'm wrong, it seems we agree that provoking such explosions would be worth it. P. >=20 > So feel free to add Reviewed-by: Christian K=C3=B6nig > . >=20 > Regards, > Christian. >=20 > > + * > > + * @sched_job: the job to run > > + * > > + * Returns: dma_fence the driver must signal once the > > hardware has > > + * completed the job ("hardware fence"). > > + * > > + * Note that the scheduler expects to 'inherit' its own > > reference to > > + * this fence from the callback. It does not invoke an > > extra > > + * dma_fence_get() on it. Consequently, this callback must > > take a > > + * reference for the scheduler, and additional ones for > > the driver's > > + * respective needs. > > =C2=A0=C2=A0 */ > > =C2=A0=C2=A0 struct dma_fence *(*run_job)(struct drm_sched_job > > *sched_job); > > =C2=A0=20 >=20