RFC: How to adjust the trace pid?

RFC: How to adjust the trace pid?

[Christian König]

Hi everybody,

in amdgpu we got the following issue which I'm seeking advise how to cleanly handle it.

We have a bunch of trace points which are related to the VM subsystem and executed in either a work item, kthread or foreign process context.

Now tracing the pid of the context which we are executing in is not really that useful, so I'm wondering if we could just overwrite the pid recorded in the trace entry?

The following patch does exactly that for the vm_grab_id() trace point, but I'm not 100% sure if that is legal or not.

Any ideas? Comments?

Thanks,
Christian.

[PATCH] drm/amdgpu: adjust the pid in the grab_id trace point

[Christian König]

Trace something useful instead of the pid of a kernel thread here.

Signed-off-by: Christian König <christian.koenig@amd.com>
---
 drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
index 5da20fc166d9..07f99ef69d91 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
@@ -228,6 +228,7 @@ TRACE_EVENT(amdgpu_vm_grab_id,
 			     ),
 
 	    TP_fast_assign(
+			   __entry->ent.pid = vm->task_info.pid;
 			   __entry->pasid = vm->pasid;
 			   __assign_str(ring, ring->name)
 			   __entry->vmid = job->vmid;
-- 
2.17.1

Re: [PATCH] drm/amdgpu: adjust the pid in the grab_id trace point

[Steven Rostedt]

On Fri, Aug 07, 2020 at 03:36:58PM +0200, Christian König wrote:
> Trace something useful instead of the pid of a kernel thread here.
> 
> Signed-off-by: Christian König <christian.koenig@amd.com>
> ---
>  drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> index 5da20fc166d9..07f99ef69d91 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> @@ -228,6 +228,7 @@ TRACE_EVENT(amdgpu_vm_grab_id,
>  			     ),
>  
>  	    TP_fast_assign(
> +			   __entry->ent.pid = vm->task_info.pid;

If the ent.pid is not the pid you are interested in for this trace event, just
add a "pid" field to the trace event and place it there. Do not modify the
generic pid that is recorded, as we would like that to be consistent for all
trace events.

The "ent.pid" turns into "common_pid" in the field, leaving "pid" free to use.
Other trace events (like sched_waking) record a pid field that is not the same
as the pid of the executing task.

The "ent.pid" should always be the pid of the task that executed the event.

-- Steve


>  			   __entry->pasid = vm->pasid;
>  			   __assign_str(ring, ring->name)
>  			   __entry->vmid = job->vmid;
> -- 
> 2.17.1

Re: [PATCH] drm/amdgpu: adjust the pid in the grab_id trace point

[Christian König]

Am 12.08.20 um 16:17 schrieb Steven Rostedt:
> On Fri, Aug 07, 2020 at 03:36:58PM +0200, Christian König wrote:
>> Trace something useful instead of the pid of a kernel thread here.
>>
>> Signed-off-by: Christian König <christian.koenig@amd.com>
>> ---
>>   drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> index 5da20fc166d9..07f99ef69d91 100644
>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>> @@ -228,6 +228,7 @@ TRACE_EVENT(amdgpu_vm_grab_id,
>>   			     ),
>>   
>>   	    TP_fast_assign(
>> +			   __entry->ent.pid = vm->task_info.pid;
> If the ent.pid is not the pid you are interested in for this trace event, just
> add a "pid" field to the trace event and place it there. Do not modify the
> generic pid that is recorded, as we would like that to be consistent for all
> trace events.

The problem my userspace guys have is that this doesn't work with 
"trace-cmd -P $pid".

But I think I can teach them how filters work :)

> The "ent.pid" turns into "common_pid" in the field, leaving "pid" free to use.
> Other trace events (like sched_waking) record a pid field that is not the same
> as the pid of the executing task.

Yes, we thought about this alternative as well.

> The "ent.pid" should always be the pid of the task that executed the event.

Why? For the case here we just execute a work item in the background for 
an userspace process.

Tracing the pid of the worker pool which executes it doesn't seem to 
make to much sense.

Thanks for the quick reply,
Christian.

>
> -- Steve
>
>
>>   			   __entry->pasid = vm->pasid;
>>   			   __assign_str(ring, ring->name)
>>   			   __entry->vmid = job->vmid;
>> -- 
>> 2.17.1

Re: [PATCH] drm/amdgpu: adjust the pid in the grab_id trace point

[Steven Rostedt]

On Wed, 12 Aug 2020 16:36:36 +0200
Christian König <ckoenig.leichtzumerken@gmail.com> wrote:

> Am 12.08.20 um 16:17 schrieb Steven Rostedt:
> > On Fri, Aug 07, 2020 at 03:36:58PM +0200, Christian König wrote:  
> >> Trace something useful instead of the pid of a kernel thread here.
> >>
> >> Signed-off-by: Christian König <christian.koenig@amd.com>
> >> ---
> >>   drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 1 +
> >>   1 file changed, 1 insertion(+)
> >>
> >> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> >> index 5da20fc166d9..07f99ef69d91 100644
> >> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> >> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
> >> @@ -228,6 +228,7 @@ TRACE_EVENT(amdgpu_vm_grab_id,
> >>   			     ),
> >>   
> >>   	    TP_fast_assign(
> >> +			   __entry->ent.pid = vm->task_info.pid;  
> > If the ent.pid is not the pid you are interested in for this trace event, just
> > add a "pid" field to the trace event and place it there. Do not modify the
> > generic pid that is recorded, as we would like that to be consistent for all
> > trace events.  
> 
> The problem my userspace guys have is that this doesn't work with 
> "trace-cmd -P $pid".
> 
> But I think I can teach them how filters work :)

Yep, trace-cmd record -e event -f "pid == $pid"

> 
> > The "ent.pid" turns into "common_pid" in the field, leaving "pid" free to use.
> > Other trace events (like sched_waking) record a pid field that is not the same
> > as the pid of the executing task.  
> 
> Yes, we thought about this alternative as well.
> 
> > The "ent.pid" should always be the pid of the task that executed the event.  
> 
> Why? For the case here we just execute a work item in the background for 
> an userspace process.
> 
> Tracing the pid of the worker pool which executes it doesn't seem to 
> make to much sense.

Maybe not for you, but it does for me. All trace events show what
happened when it happened and who executed it. I like to see what
worker threads are executing. I may filter on the worker thread, and by
changing the ent.pid, I wont see what it is doing.

That said, I think I may add a feature to a trace evnt for a special filter
to say, "test field to the set_event_pid", and if it exists in that
file to include that event in the filtered trace. This would include
sched_waking trace events as well.

That way "trace-cmd record -P $pid" will still work for your case.

-- Steve

Re: [PATCH] drm/amdgpu: adjust the pid in the grab_id trace point

[Christian König]

Am 12.08.20 um 17:19 schrieb Steven Rostedt:
> On Wed, 12 Aug 2020 16:36:36 +0200
> Christian König <ckoenig.leichtzumerken@gmail.com> wrote:
>
>> Am 12.08.20 um 16:17 schrieb Steven Rostedt:
>>> On Fri, Aug 07, 2020 at 03:36:58PM +0200, Christian König wrote:
>>>> Trace something useful instead of the pid of a kernel thread here.
>>>>
>>>> Signed-off-by: Christian König <christian.koenig@amd.com>
>>>> ---
>>>>    drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h | 1 +
>>>>    1 file changed, 1 insertion(+)
>>>>
>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>>> index 5da20fc166d9..07f99ef69d91 100644
>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_trace.h
>>>> @@ -228,6 +228,7 @@ TRACE_EVENT(amdgpu_vm_grab_id,
>>>>    			     ),
>>>>    
>>>>    	    TP_fast_assign(
>>>> +			   __entry->ent.pid = vm->task_info.pid;
>>> If the ent.pid is not the pid you are interested in for this trace event, just
>>> add a "pid" field to the trace event and place it there. Do not modify the
>>> generic pid that is recorded, as we would like that to be consistent for all
>>> trace events.
>> The problem my userspace guys have is that this doesn't work with
>> "trace-cmd -P $pid".
>>
>> But I think I can teach them how filters work :)
> Yep, trace-cmd record -e event -f "pid == $pid"
>
>>> The "ent.pid" turns into "common_pid" in the field, leaving "pid" free to use.
>>> Other trace events (like sched_waking) record a pid field that is not the same
>>> as the pid of the executing task.
>> Yes, we thought about this alternative as well.
>>
>>> The "ent.pid" should always be the pid of the task that executed the event.
>> Why? For the case here we just execute a work item in the background for
>> an userspace process.
>>
>> Tracing the pid of the worker pool which executes it doesn't seem to
>> make to much sense.
> Maybe not for you, but it does for me. All trace events show what
> happened when it happened and who executed it. I like to see what
> worker threads are executing. I may filter on the worker thread, and by
> changing the ent.pid, I wont see what it is doing.

That's enough explanation for me. Going with the separate pid field then.

Thanks,
Christian.

>
> That said, I think I may add a feature to a trace evnt for a special filter
> to say, "test field to the set_event_pid", and if it exists in that
> file to include that event in the filtered trace. This would include
> sched_waking trace events as well.
>
> That way "trace-cmd record -P $pid" will still work for your case.
>
> -- Steve

Re: RFC: How to adjust the trace pid?

[Christian König]

Ping? Daniel, Dave any opinion on this?

Christian.

Am 07.08.20 um 15:36 schrieb Christian König:
> Hi everybody,
>
> in amdgpu we got the following issue which I'm seeking advise how to cleanly handle it.
>
> We have a bunch of trace points which are related to the VM subsystem and executed in either a work item, kthread or foreign process context.
>
> Now tracing the pid of the context which we are executing in is not really that useful, so I'm wondering if we could just overwrite the pid recorded in the trace entry?
>
> The following patch does exactly that for the vm_grab_id() trace point, but I'm not 100% sure if that is legal or not.
>
> Any ideas? Comments?
>
> Thanks,
> Christian.
>
>

Re: RFC: How to adjust the trace pid?

[Daniel Vetter]

On Wed, Aug 12, 2020 at 3:42 PM Christian König
<ckoenig.leichtzumerken@gmail.com> wrote:
>
> Ping? Daniel, Dave any opinion on this?

Type patch, cc: tracing people, see what they say? tbh I have no idea,
but they have been making unhappy noises about some of the tricks
we've played in the past in i915 tracepoints. So not everything is
cool in there.

Otherwise I guess just add another tracepoint parameter to dump the
correct userspace mm.

3rd option could be to dump the current mm (since I'm assuming those
threads do kthread_use/unuse_mm to impersonate the right userspace
process correctly) in the tracepoint infrastructure too?

Cheers, Daniel

> Christian.
>
> Am 07.08.20 um 15:36 schrieb Christian König:
> > Hi everybody,
> >
> > in amdgpu we got the following issue which I'm seeking advise how to cleanly handle it.
> >
> > We have a bunch of trace points which are related to the VM subsystem and executed in either a work item, kthread or foreign process context.
> >
> > Now tracing the pid of the context which we are executing in is not really that useful, so I'm wondering if we could just overwrite the pid recorded in the trace entry?
> >
> > The following patch does exactly that for the vm_grab_id() trace point, but I'm not 100% sure if that is legal or not.
> >
> > Any ideas? Comments?
> >
> > Thanks,
> > Christian.
> >
> >
>


-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

Re: RFC: How to adjust the trace pid?

[Christian König]

Am 12.08.20 um 15:49 schrieb Daniel Vetter:
> On Wed, Aug 12, 2020 at 3:42 PM Christian König
> <ckoenig.leichtzumerken@gmail.com> wrote:
>> Ping? Daniel, Dave any opinion on this?
> Type patch, cc: tracing people, see what they say?

Adding Ingo and Steven then.

> tbh I have no idea,
> but they have been making unhappy noises about some of the tricks
> we've played in the past in i915 tracepoints. So not everything is
> cool in there.

Well that was the feedback I was looking for.

> Otherwise I guess just add another tracepoint parameter to dump the
> correct userspace mm.

Well the tracing subsystem looks like rather complicated macro magic, so 
I'm not touching that before speaking to he maintainer what approach we 
should take.

> 3rd option could be to dump the current mm (since I'm assuming those
> threads do kthread_use/unuse_mm to impersonate the right userspace
> process correctly) in the tracepoint infrastructure too?

Nope, we don't use kthread_use/unuse_mm since we don't touch the process 
which initiated the operation in any way.

This is just to improve debugging since it doesn't make much sense to 
trace the pid of the worker thread. And since we have the pid of the 
initiator around anyway using it should be trivial.

Question is rather how to do it correctly? I couldn't find much 
precedence for this.

Thanks,
Christian.

>
> Cheers, Daniel
>
>> Christian.
>>
>> Am 07.08.20 um 15:36 schrieb Christian König:
>>> Hi everybody,
>>>
>>> in amdgpu we got the following issue which I'm seeking advise how to cleanly handle it.
>>>
>>> We have a bunch of trace points which are related to the VM subsystem and executed in either a work item, kthread or foreign process context.
>>>
>>> Now tracing the pid of the context which we are executing in is not really that useful, so I'm wondering if we could just overwrite the pid recorded in the trace entry?
>>>
>>> The following patch does exactly that for the vm_grab_id() trace point, but I'm not 100% sure if that is legal or not.
>>>
>>> Any ideas? Comments?
>>>
>>> Thanks,
>>> Christian.
>>>
>>>
>