Re: [PATCH 4/5] media: iris: reject open() when the session limit is reached

[Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>]

On 7/30/26 5:26 PM, Dmitry Baryshkov wrote:
> iris_add_session() silently skips adding the instance to core->instances
> once max_session_count is reached, but returns void, so iris_open()
> continues as if it succeeded and hands a file descriptor back to
> userspace. As the instance is not on core->instances, firmware responses
> for it are dropped and every subsequent ioctl times out.
> 
> Make iris_add_session() return an error when the limit is reached and
> fail iris_open() accordingly, freeing the partially initialised instance.
> 
> Fixes: 38fc8beaba55 ("media: iris: implement reqbuf ioctl with vb2_queue_setup")
> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> Assisted-by: Claude:claude-opus-4-8
> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> ---
>  drivers/media/platform/qcom/iris/iris_vidc.c | 14 ++++++++++++--
>  1 file changed, 12 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/platform/qcom/iris/iris_vidc.c b/drivers/media/platform/qcom/iris/iris_vidc.c
> index 14d63dc76c9b..b68b98f02e26 100644
> --- a/drivers/media/platform/qcom/iris/iris_vidc.c
> +++ b/drivers/media/platform/qcom/iris/iris_vidc.c
> @@ -39,11 +39,12 @@ static void iris_v4l2_fh_deinit(struct iris_inst *inst, struct file *filp)
>  	v4l2_fh_exit(&inst->fh);
>  }
>  
> -static void iris_add_session(struct iris_inst *inst)
> +static int iris_add_session(struct iris_inst *inst)
>  {
>  	struct iris_core *core = inst->core;
>  	struct iris_inst *iter;
>  	u32 count = 0;
> +	int ret = 0;
>  
>  	mutex_lock(&core->lock);
>  
> @@ -52,8 +53,12 @@ static void iris_add_session(struct iris_inst *inst)
>  
>  	if (count < core->iris_platform_data->max_session_count)
>  		list_add_tail(&inst->list, &core->instances);
> +	else
> +		ret = -EBUSY;
>  
>  	mutex_unlock(&core->lock);
> +
> +	return ret;

Since you already have to resend.. nit:

guard(mutex)(&core->lock)

if (count >= core->iris_platform_data->max_session_count)
	return -EBUSY

list_add_tail

return 0;

Konrad