From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-qt1-f174.google.com (mail-qt1-f174.google.com [209.85.160.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 20BAB372B5E for ; Thu, 16 Jul 2026 04:11:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=pass smtp.client-ip=209.85.160.174 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784175109; cv=pass; b=kbIaYFQ/a1XSWwH3QaZtkCScG6OlMAS3ghzlSoN+fiNQ0Owx9eGs36zMDplEXN3sLf/9v5hvCITgCiLgGpTCqYT+zXZ+6mF3wiTWHPgiD3pgzpU2VUSVAG4YX/JvXy//M4cZW3uVzxJVQFNZvZ+OOAuXy60YhnhPp+FiFSEs0vM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784175109; c=relaxed/simple; bh=SnctVqNMHifyjiQrx5QG8yoTRdoh14J2ybAjo9hwyYg=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=uyCdbkosTlDEFSylZgZUbnxFnu+YNvwBPJOhcHwfk2e3BpVLgTWl195op7ELm2UzUtStbXJ5VrKBCv0Wd5NZHiS+1QPz8Qq/Qajqn/i+9b8AQjUpLr5gRTGOypjxBM9RVXaKziu5w2rzw7F/3CWa+dzcSKGz223+c5SxFl+dDuM= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PKp2V54w; arc=pass smtp.client-ip=209.85.160.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PKp2V54w" Received: by mail-qt1-f174.google.com with SMTP id d75a77b69052e-51c01c79467so160111cf.0 for ; Wed, 15 Jul 2026 21:11:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1784175107; cv=none; d=google.com; s=arc-20260327; b=Rt0jP6mBoFcVXMZyDMq6ujGdrtqPaQ/JnqP8a4kg0dvhMcz1eUi7wiJXkfUFEvZHyd ysF8UdCZ44aNm88Rn/tThwqiigdGGXrHmCqbI5IJVETsVO4iaJBSQiGIBu68B8TnlzcC /Dx2icscJ5E5ZgtQTuIusJ5LNHR45rzOvCRILSHttJifpxgT8CAVbGQIc9H0EeXIAIQh nJygPwCaYMPwadPmcYvRAuYtEwpqBYtJKZKVOO0u/HOMd8Q5IDIZmj9jFoaJNQBtd/Vz cVGfafdHofZv+e9qD7x3pBdKbaEfwKGY5+YGbMkDW6UXXLp4VONuTeNuQNbOS6Eha220 LxbA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:dkim-signature; bh=NrULQ860uqutCMctJCUNcuP6jCmEncIkfJ8w3bYOK54=; fh=uI0tZj0rqAsXYo0ssYwGBI6LdtBnuKyI4LJLnW/9dHk=; b=IH/YNdmNudmJddQCwIS9BLhR/Zqd01h9FCG+Hejndk4a488sPX/D+E8Jo1+xCAD2rY kavhxc+40JlQyvtGp2MlZpQAB95fCZyClmGWvoC/eEbLQj9sIntdbO58L+czvxRvsl4W pcaXhyibciUX+lhjqWU0s4etKZaXZjKao9iKzndmUugUdlKdambyZ3M89Jy1r4t2PCgG /cSrfFVuM7lw7mUbNZp9dMlQdvK9DMqXfJwjVKgweZmEpTfm7zS3E3YYC3XcuGxzufyb W6RZ7llOjvTwBdf8rW1RsY+ee42DY9DCKt0CjORGCxE3+QkFn3MnvF1/61f2jt17nd70 tcyA==; darn=vger.kernel.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1784175107; x=1784779907; darn=vger.kernel.org; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:from:in-reply-to:references:mime-version:from:to:cc:subject :date:message-id:reply-to:content-type; bh=NrULQ860uqutCMctJCUNcuP6jCmEncIkfJ8w3bYOK54=; b=PKp2V54ws5Xp62jvyp390c4NzDm+EM2Q51egx698AYj5YPnvZxPcmz21zQEdOb5s9g q12nS/2itHhH4EZ8lWDHwZwGyHFmqM3WsuBxdIZddlXNnoqGRVKsgPg4esfVElWua5iZ 3aPrCMAM5o1opNU9tlznauVoL9P8k277kJVixhnOVCiu/jGloNjgFq1IMXru/4YaxGR0 NohkgawZkBnpy8e5REqQSxHFhagbBfqlpqgcjbhoUtGsIqDRJVNZjTvM9GA1BsNdtC77 BRCXqxW0ErD0FyKI5qcCmTzvmrpNvAlCaGs8XELcPQCLXmLLzRMEmeEZTE4oHHw37zq5 27Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784175107; x=1784779907; h=content-transfer-encoding:content-type:cc:to:subject:message-id :date:from:in-reply-to:references:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=NrULQ860uqutCMctJCUNcuP6jCmEncIkfJ8w3bYOK54=; b=UgjM+y2fZsglU9iGHqhS5hF6KuRkMm58svcVqLwRpaBoCSmAwk2ocHc2c6+oS/2X3o 2b9uKc+70nfVE9e3KYKCOsFZuUgAbFbi6vghX+n13RDQHV7YcbU4UK5KGfzILVCWTior KZdkMSO2caJq+hgW1Q3IyrhxocQ11SagqMTThTyAl7ueBSD/F1vnPXQPSZ0mPOaD9gVY hkNDL0Z8tK0pmo9lCIG2mIQP0doFJ3KqNoUS7GZxqjStR0pppjfMuq07v8aXBdstwQ+j HnXDJvhCB0KZvUPTbwr8u6pjk7JBhQr0KJH/x62wdGaQ8ZJyLzQRoE7q3pwOTLuGWytP EvaQ== X-Forwarded-Encrypted: i=1; AHgh+RoCullgZzRwVnel+3JMkdsdMt4aezdaGORifCM3tIMSrZip16wc+yE8ev4DODot4WZapEKY2QfxTUpyGX8=@vger.kernel.org X-Gm-Message-State: AOJu0YwB+nYdafuZumIzt4UNU0fd4RbvABiYhStuzEm6Dk3Yfm+fy7O9 aduiS8ZxhmFkcaTciMsmxbXrq53rgmXgxju6+hbwJFY7T6KbKN8YcfDW1sP/R7aUEWW5nkswpkL Yy3rw2v3AYSVqIWxj/ZS+JWeSuhOHP4zmCliJ0C88 X-Gm-Gg: AfdE7cmNhwxsuoaFpvTJkpYPanUAhmIdZBO+PxMj1eVPtGYEj/cijximZfzpG/U7U2p 8BxV7prgE14789TM/vPzWFLbHTQluHPEMT7Psy85kap+/4bg0jRlcsXUuOv2Dqu5iSk5SLyfV9t 3oZNQCFn+2moSS9ViWRWwU21FNzu2crawLGDqicXnMv4WB5PqxNRgj65Q/4HlhIleUMXqcSjdXC yimIV9ABQh1Bf1LAtGdh96PyxT4B8fx8OpVV1wO109sUt2GiTGYbf5x6dz6 X-Received: by 2002:a05:622a:2c7:b0:51c:ad5:93be with SMTP id d75a77b69052e-51f7e70546dmr4830331cf.8.1784175106282; Wed, 15 Jul 2026 21:11:46 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20260715-b4-objext_split-v1-0-9a49c4ccf4c3@kernel.org> <20260715-b4-objext_split-v1-8-9a49c4ccf4c3@kernel.org> In-Reply-To: <20260715-b4-objext_split-v1-8-9a49c4ccf4c3@kernel.org> From: Suren Baghdasaryan Date: Wed, 15 Jul 2026 21:11:34 -0700 X-Gm-Features: AUfX_mzmBxfVFHGAloT5zOeFHiT-qyP82qT-DjMIIzg-ZYl4231yZILA88Rx53s Message-ID: Subject: Re: [PATCH RFC 08/12] mm/slab: change struct slabobj_ext to a union To: "Vlastimil Babka (SUSE)" Cc: Harry Yoo , Hao Li , Shakeel Butt , Alexander Potapenko , Marco Elver , Andrew Morton , Christoph Lameter , David Rientjes , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, Jul 15, 2026 at 3:11=E2=80=AFAM Vlastimil Babka (SUSE) wrote: > > Currently, struct slabobj_ext can hold both objcg pointer and > codetag_ref (when both are compile-enabled) and there is an array of as > many slabobj_ext instances as there are objects in a slab. > > This makes the layout fixed so even if codetag_ref is unused (because > memory allocation profiling is disabled), the space for them is > allocated and wasted. Similarly, some caches (currently kmalloc_normal) > do not ever need objcg pointers, leading to wasted memory with memory > allocation profiling enabled. > > To make this more flexible, change the layout so that struct slabobj_ext > becomes an union of objcg pointer and codetag_ref (to ensure uniform > size; in practice both are the same size anyway). The slabobj_ext array > then can have twice as many elements as before. For cache locality > purposes, the effective memory layout is unchanged, so objcg and codetag > ref for a given object are still adjacent. > > static_obj_ext_size() returns the effective size of (0-2) struct > slabobj_ext's, depending on the config. slab_obj_ext_size() is currently > static as well, but takes a slab pointer so it can be made dynamic > later. Replace all sizeof(slabobj_ext) usage with these. > > No functional change intended, the layout is still effectively static. > > Signed-off-by: Vlastimil Babka (SUSE) This is much cleaner than what I was preparing. Nicely done! Reviewed-by: Suren Baghdasaryan > --- > mm/slab.h | 41 +++++++++++++++++++++++++++++++++-------- > mm/slub.c | 17 +++++++++-------- > 2 files changed, 42 insertions(+), 16 deletions(-) > > diff --git a/mm/slab.h b/mm/slab.h > index 3ad9777ad600..359ab8caf61e 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -554,14 +554,34 @@ static inline bool need_kmalloc_no_objext(void) > * if MEMCG_DATA_OBJEXTS is set. > */ > struct slabobj_ext { Perhaps we should add a comment here stating that every element of this union should be pointer-sized? > + union { > #ifdef CONFIG_MEMCG > - struct obj_cgroup *_objcg; > + struct obj_cgroup *_objcg; > #endif > #ifdef CONFIG_MEM_ALLOC_PROFILING > - union codetag_ref _ctref; > + union codetag_ref _ctref; > #endif > + }; > } __aligned(8); > > +static inline size_t static_obj_ext_size(void) > +{ > + size_t sz =3D 0; > + > + if (IS_ENABLED(CONFIG_MEMCG)) > + sz +=3D 1; > + > + if (IS_ENABLED(CONFIG_MEM_ALLOC_PROFILING)) > + sz +=3D 1; > + > + return sizeof(struct slabobj_ext) * sz; > +} > + > +static inline size_t slab_obj_ext_size(struct slab *slab) > +{ > + return static_obj_ext_size(); > +} > + > #ifdef CONFIG_SLAB_OBJ_EXT > > /* > @@ -650,17 +670,18 @@ slab_obj_ext(struct kmem_cache *s, struct slab *sla= b, unsigned long obj_exts, > { > struct slabobj_ext *obj_ext; > unsigned int index; > + unsigned int stride; > > VM_WARN_ON_ONCE(obj_exts !=3D slab_obj_exts(slab)); > > index =3D obj_to_index(s, slab, obj); > > - if (!obj_exts_in_object(slab)) { > - obj_ext =3D ((struct slabobj_ext *)obj_exts) + index; > - } else { > - unsigned int stride =3D s->size; > - obj_ext =3D (struct slabobj_ext *)(obj_exts + index * str= ide); > - } > + if (!obj_exts_in_object(slab)) > + stride =3D slab_obj_ext_size(slab); > + else > + stride =3D s->size; > + > + obj_ext =3D (struct slabobj_ext *)(obj_exts + index * stride); > > return kasan_reset_tag(obj_ext); > } > @@ -668,6 +689,7 @@ slab_obj_ext(struct kmem_cache *s, struct slab *slab,= unsigned long obj_exts, > #ifdef CONFIG_MEMCG > static inline struct obj_cgroup **slab_obj_ext_objcgp(struct slabobj_ext= *obj_ext) > { > + /* if objcg exists, it's first, so we don't need to do anything *= / > return &obj_ext->_objcg; > } > #endif > @@ -676,6 +698,9 @@ static inline struct obj_cgroup **slab_obj_ext_objcgp= (struct slabobj_ext *obj_ex > static inline union codetag_ref * > slab_obj_ext_codetag_ref(struct slab *slab, struct slabobj_ext *obj_ext) > { > + if (IS_ENABLED(CONFIG_MEMCG)) > + obj_ext +=3D 1; > + > return &obj_ext->_ctref; > } > #endif > diff --git a/mm/slub.c b/mm/slub.c > index 98a14e5842a2..dd15af8abd62 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -803,7 +803,7 @@ static inline bool need_slab_obj_exts(struct kmem_cac= he *s) > > static inline unsigned int obj_exts_size_in_slab(struct slab *slab) > { > - return sizeof(struct slabobj_ext) * slab->objects; > + return slab_obj_ext_size(slab) * slab->objects; > } > > static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s= , > @@ -1199,7 +1199,7 @@ static void print_trailer(struct kmem_cache *s, str= uct slab *slab, u8 *p) > off +=3D kasan_metadata_size(s, false); > > if (obj_exts_in_object(slab)) > - off +=3D sizeof(struct slabobj_ext); > + off +=3D slab_obj_ext_size(slab); > > if (off !=3D size_from_object(s)) > /* Beginning of the filler is the free pointer */ > @@ -1404,7 +1404,7 @@ static int check_pad_bytes(struct kmem_cache *s, st= ruct slab *slab, u8 *p) > off +=3D kasan_metadata_size(s, false); > > if (obj_exts_in_object(slab)) > - off +=3D sizeof(struct slabobj_ext); > + off +=3D slab_obj_ext_size(slab); > > if (size_from_object(s) =3D=3D off) > return 1; > @@ -2089,6 +2089,8 @@ static inline bool mark_failed_objexts_alloc(struct= slab *slab) > static inline void handle_failed_objexts_alloc(struct slab *slab, > unsigned long obj_exts, struct slabobj_ext *vec) > { > + unsigned int stride =3D slab_obj_ext_size(slab) / sizeof(*vec); > + > /* > * If vector previously failed to allocate then we have live > * objects with no tag reference. Mark all references in this > @@ -2101,7 +2103,7 @@ static inline void handle_failed_objexts_alloc(stru= ct slab *slab, > union codetag_ref *ref =3D slab_obj_ext_codetag_ref(slab,= vec); > > set_codetag_empty(ref); > - vec++; > + vec +=3D stride; > } > } > > @@ -2127,7 +2129,7 @@ int alloc_slab_obj_exts(struct slab *slab, struct k= mem_cache *s, > unsigned long new_exts; > unsigned long old_exts; > struct slabobj_ext *vec; > - size_t sz =3D sizeof(struct slabobj_ext) * slab->objects; > + size_t sz =3D slab_obj_ext_size(slab) * slab->objects; > > gfp &=3D ~OBJCGS_CLEAR_MASK; > /* > @@ -2270,8 +2272,7 @@ static void alloc_slab_obj_exts_early(struct kmem_c= ache *s, struct slab *slab) > > get_slab_obj_exts(obj_exts); > for_each_object(addr, s, slab_address(slab), slab->object= s) > - memset(kasan_reset_tag(addr) + offset, 0, > - sizeof(struct slabobj_ext)); > + memset(kasan_reset_tag(addr) + offset, 0, slab_ob= j_ext_size(slab)); > put_slab_obj_exts(obj_exts); > > #ifdef CONFIG_MEMCG > @@ -7930,7 +7931,7 @@ static int calculate_sizes(struct kmem_cache_args *= args, struct kmem_cache *s) > aligned_size =3D ALIGN(size, s->align); > #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT) > if (slab_args_unmergeable(args, s->flags) && > - (aligned_size - size >=3D sizeof(struct slabobj_e= xt))) > + (aligned_size - size >=3D static_obj_ext_size())) > s->flags |=3D SLAB_OBJ_EXT_IN_OBJ; > #endif > size =3D aligned_size; > > -- > 2.55.0 >