From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S269770AbUJALv5 (ORCPT ); Fri, 1 Oct 2004 07:51:57 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S269767AbUJALv5 (ORCPT ); Fri, 1 Oct 2004 07:51:57 -0400 Received: from open.hands.com ([195.224.53.39]:23233 "EHLO open.hands.com") by vger.kernel.org with ESMTP id S269764AbUJALvO (ORCPT ); Fri, 1 Oct 2004 07:51:14 -0400 Date: Fri, 1 Oct 2004 13:02:22 +0100 From: Luke Kenneth Casson Leighton To: linux-kernel@vger.kernel.org Subject: making an in-memory hashing table ["name" -> ino_t] with thousands of entries Message-ID: <20041001120222.GA8507@lkcl.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.5.1+cvs20040105i X-hands-com-MailScanner: Found to be clean X-hands-com-MailScanner-SpamScore: s X-MailScanner-From: lkcl@lkcl.net Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org dear kernel hackers, i seek advice on how to do an in-kernel lookup table capable of storing thousands, potentially hundreds of thousands, of entries. the reason for this is to move the fuse userspace inode lookup tables into the kernel. fuse userspace servers have their own in-memory database of unique inode numbers which represent the file names, and there is a communication mechanism between userspace and kernelspace that transfers those inode numbers, amongst other things. is there any _sane_ way to do this or should i leave the inode lookup table where it presently is - in userspace? bearing in mind that for every file accessed via a fuse filesystem, a cache entry is created, and therefore the number of entries could potentially run into hundreds of thousands of entries. l. -- -- Truth, honesty and respect are rare commodities that all spring from the same well: Love. If you love yourself and everyone and everything around you, funnily and coincidentally enough, life gets a lot better. -- lkcl.net
lkcl@lkcl.net
From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S269764AbUJAMaZ (ORCPT ); Fri, 1 Oct 2004 08:30:25 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S269769AbUJAMaZ (ORCPT ); Fri, 1 Oct 2004 08:30:25 -0400 Received: from faui3es.informatik.uni-erlangen.de ([131.188.33.16]:60632 "EHLO faui3es.informatik.uni-erlangen.de") by vger.kernel.org with ESMTP id S269764AbUJAMaW (ORCPT ); Fri, 1 Oct 2004 08:30:22 -0400 Date: Fri, 1 Oct 2004 14:30:19 +0200 From: Martin Waitz To: Luke Kenneth Casson Leighton Cc: linux-kernel@vger.kernel.org Subject: Re: making an in-memory hashing table ["name" -> ino_t] with thousands of entries Message-ID: <20041001123019.GC4072@admingilde.org> Mail-Followup-To: Luke Kenneth Casson Leighton , linux-kernel@vger.kernel.org References: <20041001120222.GA8507@lkcl.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uh9ZiVrAOUUm9fzH" Content-Disposition: inline In-Reply-To: <20041001120222.GA8507@lkcl.net> User-Agent: Mutt/1.3.28i X-Habeas-SWE-1: winter into spring X-Habeas-SWE-2: brightly anticipated X-Habeas-SWE-3: like Habeas SWE (tm) X-Habeas-SWE-4: Copyright 2002 Habeas (tm) X-Habeas-SWE-5: Sender Warranted Email (SWE) (tm). The sender of this X-Habeas-SWE-6: email in exchange for a license for this Habeas X-Habeas-SWE-7: warrant mark warrants that this is a Habeas Compliant X-Habeas-SWE-8: Message (HCM) and not spam. Please report use of this X-Habeas-SWE-9: mark in spam to . X-PGP-Fingerprint: B21B 5755 9684 5489 7577 001A 8FF1 1AC5 DFE8 0FB2 Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org --uh9ZiVrAOUUm9fzH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable hi :) On Fri, Oct 01, 2004 at 01:02:22PM +0100, Luke Kenneth Casson Leighton wrot= e: > bearing in mind that for every file accessed via a fuse > filesystem, a cache entry is created, and therefore the number > of entries could potentially run into hundreds of thousands > of entries. but these can be cleaned if needed. why do you have to move all inodes into the kernel when the kernel already cache those that are really needed? (perhaps I'm misunderstandig fuses role here) --=20 Martin Waitz --uh9ZiVrAOUUm9fzH Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQFBXU3aj/Eaxd/oD7IRApIMAJ4s8LUQgxFeetugJPX0xlTV0K3mdwCfV2pM tTXrKpc7dovcQ7PGzHB17xg= =t1L5 -----END PGP SIGNATURE----- --uh9ZiVrAOUUm9fzH-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S269785AbUJANV7 (ORCPT ); Fri, 1 Oct 2004 09:21:59 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S269787AbUJANV7 (ORCPT ); Fri, 1 Oct 2004 09:21:59 -0400 Received: from open.hands.com ([195.224.53.39]:42179 "EHLO open.hands.com") by vger.kernel.org with ESMTP id S269785AbUJANVz (ORCPT ); Fri, 1 Oct 2004 09:21:55 -0400 Date: Fri, 1 Oct 2004 14:33:02 +0100 From: Luke Kenneth Casson Leighton To: linux-kernel@vger.kernel.org Subject: Re: making an in-memory hashing table ["name" -> ino_t] with thousands of entries Message-ID: <20041001133302.GB8507@lkcl.net> References: <20041001120222.GA8507@lkcl.net> <20041001123019.GC4072@admingilde.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041001123019.GC4072@admingilde.org> User-Agent: Mutt/1.5.5.1+cvs20040105i X-hands-com-MailScanner: Found to be clean X-hands-com-MailScanner-SpamScore: s X-MailScanner-From: lkcl@lkcl.net Sender: linux-kernel-owner@vger.kernel.org X-Mailing-List: linux-kernel@vger.kernel.org On Fri, Oct 01, 2004 at 02:30:19PM +0200, Martin Waitz wrote: > hi :) hello martin: thanks for responding. > On Fri, Oct 01, 2004 at 01:02:22PM +0100, Luke Kenneth Casson Leighton wrote: > > bearing in mind that for every file accessed via a fuse > > filesystem, a cache entry is created, and therefore the number > > of entries could potentially run into hundreds of thousands > > of entries. > > but these can be cleaned if needed. > > why do you have to move all inodes into the kernel when the kernel > already cache those that are really needed? > (perhaps I'm misunderstandig fuses role here) fuse is a userspace filesystem. it can be absolutely anything. however, the author of fuse, in order to simplify the userspace interface, manages the userspace-inode <-> userspace-fullpathnames in a library. for the sake of argument, let's call this the fuse-inode-cache. entries in the fuse-inode-cache must be persistent for as long as a filename exists on the userspace file system. also, if an entry happens not to exist at the time a lookup is done, then an entry into the cache is created, associating that full path name with a newly created unique inode number (in the userspace fuse-inode-cache). then, that information (the unique inode number) is communicated _back_ to the fuse module, along with the stat information (yes, a userspace lstat is done as well), and the kernel's inode structure is updated from that information. anyway: for various reasons, inside the kernel, it must be possible to do a lookup from the full path name to the inode number. smbfs suffers from exactly the same problem: inodes don't exist on the SMB protocol. hm, i wonder if i should just rip stacks of code from the smbfs kernel module? i note with interest the existence of a function smb_refresh_inode() which gets passed a dentry. then i can do a getattr and go from there. oh, man, this is hair-raising stuff. l. -- -- Truth, honesty and respect are rare commodities that all spring from the same well: Love. If you love yourself and everyone and everything around you, funnily and coincidentally enough, life gets a lot better. -- lkcl.net
lkcl@lkcl.net