Re: ConfigFS + Target Mode Engine API discussion

Re: ConfigFS + Target Mode Engine API discussion

[Nicholas A. Bellinger]

On Thu, 2008-09-11 at 01:58 -0700, Nicholas A. Bellinger wrote:
> On Wed, 2008-09-10 at 23:38 -0700, Joel Becker wrote:
> > On Wed, Sep 10, 2008 at 09:29:18PM -0700, Nicholas A. Bellinger wrote:
> > > On Wed, 2008-09-10 at 19:13 -0700, Joel Becker wrote:
> > > > On Wed, Sep 10, 2008 at 06:42:46PM -0700, Nicholas A. Bellinger wrote:
> > > > > So I am thinking about the following question:  What would be the
> > > > > preferred method for calling ct_group_ops->make_group() in order to
> > > > > create the $CONFIGFS/target/$FABRIC struct config_item directly from
> > > > > target_fabric_configfs_register() call?  From there, the config group
> > > > > hanging off $CONFIGFS/target/$FABRIC will be fabric dependent and
> > > > > providing their own groups, items, depends, from the passed *fabric_cit.
> > > > > How do I "simulate" a mkdir(2) configfs -> make_group() call coming from
> > > > > the fabric module itself..?   This would be assuming that both mkdir(2)
> > > > 
> > > > 	That's precisely what you don't do with configfs.  It's a
> > > > defined "not to be done" thing.  So there's no preferred way, there's no
> > > > way at all.
> > > > 	What you want do to is drive this from mkdir().  The
> > > > make_group() will look up the sub-module it needs and return the
> > > > appropriate item.
> > > > 
> > > 
> > > Whew, good thing I asked about this case first..  :-)
> > 
> > 	Can you give me a more complete description of what you're
> > trying to do?  that way I can maybe help with some suggestions.
> > 
> 
> Sure, the process to get the things up and running would look like:
> 
> # Load in the generic target's configfs infrastructure
> modprobe target_core_configfs
> 
> *) $CONFIGFS/target is created with configfs_register_subsystem()
> 
> *) $CONFIGFS/target/core is created as a default group under struct
> configfs_subsystem->su_group to interact with generic target mode
> engine's storage objects (eg: SCSI HCTL referenced devices, LVM UUID, MD
> UUID). 
> 
> ...........
> 
>  # Load LIO-Target using function symbols from target_core_configfs
> modprobe iscsi_target_mod
> 
> module_init() from iscsi_target_mod (eg: the fabric module) calls
> target_fabric_configfs_register() in target_core_configfs.ko, which is a
> small wrapper for calling sys_mkdir($CONFIGFS/target/$FABRIC) to kick
> off the struct config_group_operations->make_group() to create a struct
> config_item for /sys/kernel/config/target/iscsi (or whatever the fabric
> is called).
> 
> config_item_get() is also called in target_fabric_configfs_register()
> and returns the newly allocated struct config_item to
> iscsi_target_mod.ko code to then create new struct config_groups for
> iSCSI target fabric specific abstractions and LUN mappings between
> $CONFIGFS/target/core/$STORAGE_OBJECT and Fabric dependent code. I think
> the latter would be done with symlinks between $FABRIC <-> Generic
> Target Core Storage Object.
> 
> Anyways, knowing that make_group() and drop_item() will *NEVER* be
> called internally clears up alot for me.  I will keep working on this
> setup and let you know if I run into any more head scratchers.
> 
> Thanks again Joel!
> 

Hi Joel,

After some more thought and a few hours of effort, I was able to get
some code up using the model above between a new target_core_configfs.ko
and iscsi_target_mod.ko (LIO-Target):

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commit;h=1a48fb089e1ee1cff5ef02b401d365fdf206cf69

The one symbol that I required out of fs/namei.c for vfs_mkdir() that
was not available was lookup_hash(), and the symbol is exported in the
commit so that
drivers/lio-core/transport_core_configfs.c:do_configfs_rmdir() works as
expected.  There is another version of do_configfs_rmdir() that uses the
exported lookup_one_len(), but that one is not working just quite yet.
Perhaps there is value in lookup_hash() being defined as
EXPORT_SYMBOL_GPL().

Now that the basic configfs structure between a generic engine <->
fabric is in place, the next steps will be to enable the LIO-Target  and
LIO-Core IOCTLs into make_group() and drop_item() calls.

Have a look and let me know what you think..  I will ping you when the
first pieces of LIO-Target functionality are up and running..

:-)

--nab

Re: ConfigFS + Target Mode Engine API discussion

[Nicholas A. Bellinger]

On Fri, 2008-09-12 at 09:24 -0700, Nicholas A. Bellinger wrote:
> On Thu, 2008-09-11 at 01:58 -0700, Nicholas A. Bellinger wrote:
> > On Wed, 2008-09-10 at 23:38 -0700, Joel Becker wrote:
> > > On Wed, Sep 10, 2008 at 09:29:18PM -0700, Nicholas A. Bellinger wrote:
> > > > On Wed, 2008-09-10 at 19:13 -0700, Joel Becker wrote:
> > > > > On Wed, Sep 10, 2008 at 06:42:46PM -0700, Nicholas A. Bellinger wrote:
> > > > > > So I am thinking about the following question:  What would be the
> > > > > > preferred method for calling ct_group_ops->make_group() in order to
> > > > > > create the $CONFIGFS/target/$FABRIC struct config_item directly from
> > > > > > target_fabric_configfs_register() call?  From there, the config group
> > > > > > hanging off $CONFIGFS/target/$FABRIC will be fabric dependent and
> > > > > > providing their own groups, items, depends, from the passed *fabric_cit.
> > > > > > How do I "simulate" a mkdir(2) configfs -> make_group() call coming from
> > > > > > the fabric module itself..?   This would be assuming that both mkdir(2)
> > > > > 
> > > > > 	That's precisely what you don't do with configfs.  It's a
> > > > > defined "not to be done" thing.  So there's no preferred way, there's no
> > > > > way at all.
> > > > > 	What you want do to is drive this from mkdir().  The
> > > > > make_group() will look up the sub-module it needs and return the
> > > > > appropriate item.
> > > > > 
> > > > 
> > > > Whew, good thing I asked about this case first..  :-)
> > > 
> > > 	Can you give me a more complete description of what you're
> > > trying to do?  that way I can maybe help with some suggestions.
> > > 
> > 
> > Sure, the process to get the things up and running would look like:
> > 
> > # Load in the generic target's configfs infrastructure
> > modprobe target_core_configfs
> > 
> > *) $CONFIGFS/target is created with configfs_register_subsystem()
> > 
> > *) $CONFIGFS/target/core is created as a default group under struct
> > configfs_subsystem->su_group to interact with generic target mode
> > engine's storage objects (eg: SCSI HCTL referenced devices, LVM UUID, MD
> > UUID). 
> > 
> > ...........
> > 
> >  # Load LIO-Target using function symbols from target_core_configfs
> > modprobe iscsi_target_mod
> > 
> > module_init() from iscsi_target_mod (eg: the fabric module) calls
> > target_fabric_configfs_register() in target_core_configfs.ko, which is a
> > small wrapper for calling sys_mkdir($CONFIGFS/target/$FABRIC) to kick
> > off the struct config_group_operations->make_group() to create a struct
> > config_item for /sys/kernel/config/target/iscsi (or whatever the fabric
> > is called).
> > 
> > config_item_get() is also called in target_fabric_configfs_register()
> > and returns the newly allocated struct config_item to
> > iscsi_target_mod.ko code to then create new struct config_groups for
> > iSCSI target fabric specific abstractions and LUN mappings between
> > $CONFIGFS/target/core/$STORAGE_OBJECT and Fabric dependent code. I think
> > the latter would be done with symlinks between $FABRIC <-> Generic
> > Target Core Storage Object.
> > 
> > Anyways, knowing that make_group() and drop_item() will *NEVER* be
> > called internally clears up alot for me.  I will keep working on this
> > setup and let you know if I run into any more head scratchers.
> > 
> > Thanks again Joel!
> > 
> 
> Hi Joel,
> 
> After some more thought and a few hours of effort, I was able to get
> some code up using the model above between a new target_core_configfs.ko
> and iscsi_target_mod.ko (LIO-Target):
> 
> http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commit;h=1a48fb089e1ee1cff5ef02b401d365fdf206cf69
> 
> The one symbol that I required out of fs/namei.c for vfs_mkdir() that

This should have been vfs_rmdir()..  The commit requires no unexported
functions for vfs_mkdir().

--nab

> was not available was lookup_hash(), and the symbol is exported in the
> commit so that
> drivers/lio-core/transport_core_configfs.c:do_configfs_rmdir() works as
> expected.  There is another version of do_configfs_rmdir() that uses the
> exported lookup_one_len(), but that one is not working just quite yet.
> Perhaps there is value in lookup_hash() being defined as
> EXPORT_SYMBOL_GPL().
> 
> Now that the basic configfs structure between a generic engine <->
> fabric is in place, the next steps will be to enable the LIO-Target  and
> LIO-Core IOCTLs into make_group() and drop_item() calls.
> 
> Have a look and let me know what you think..  I will ping you when the
> first pieces of LIO-Target functionality are up and running..
> 
> :-)
> 
> --nab
> 
> 
> 
> --~--~---------~--~----~------------~-------~--~----~
> You received this message because you are subscribed to the Google Groups "Linux-iSCSI.org Target Development" group.
> To post to this group, send email to linux-iscsi-target-dev@googlegroups.com
> To unsubscribe from this group, send email to linux-iscsi-target-dev+unsubscribe@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/linux-iscsi-target-dev?hl=en
> -~----------~----~----~----~------~----~------~--~---
> 

Re: ConfigFS + Target Mode Engine API discussion

[Nicholas A. Bellinger]

On Sat, 2008-09-13 at 12:22 -0700, Nicholas A. Bellinger wrote:
> On Fri, 2008-09-12 at 21:49 -0700, Joel Becker wrote:
> > On Fri, Sep 12, 2008 at 03:27:57PM -0700, Nicholas A. Bellinger wrote:
> > > On Fri, 2008-09-12 at 13:15 -0700, Joel Becker wrote:
> > > > 	No, you don't do this.  Like I said, configfs clients let
> > > > userspace create and destroy items.  Not kernelspace.  Do not call
> > > > sys_mkdir()/sys_rmdir() from your modules.
> > > 
> > > Well, the startup 'fabric registration' is only case where I figured
> > > vfs_mkdir($CONFIGFS/target/$FABRIC) being called in order to have the
> > > fabric struct config_item appear at modprobe $FABRIC_MOD *BEFORE* the
> > > user did anything would be beneficial at all.   If vfs_mkdir() was not
> > > called from transport_fabric_register_configfs(), it only means user
> > > would have call mkdir $CONFIGFS/target/$FABRIC and/or mkdir -p
> > > $CONFIGFS/target/$FABRIC/endpoint to kick off one common make_group()
> > > for $FABRIC (that lives in target_core_configfs.c) and then make_group()
> > > that likes inside of $FABRIC_MOD who's parent is $CONFIGFS/target.  In
> > > LIO-Target's case, this would be creating a new iSCSI Qualified Name
> > > with mkdir $CONFIGFS/target/iscsi/iqn.superturobdiskarry
> > 
> > 	I'm still totally unclear as to why $FABRIC has to live under
> > target/.
> 
> Well, I figured that since $FABRIC would be depending upon LUN mappings
> to target/core/$STORAGE_OBJECT struct config_items , it would make sense
> sense for $FABRIC to appear under a common target mode engine..
> 
> Having $CONFIGFS/target/core for storage objects and
> $CONFIG/$FABRIC/endpoint/lun_0 and setting up symlinks between the two
> entities is what I think you have in mind.  This setup you mention may
> have other advantages/disadvantages when it goes to a generic target
> mode, but I have been focused on getting the initial setup (see below..)
> up and running, sooo.. :-) Obviously if things can be done easier I am
> all for it.  Also I think for debugging and [PROC,IOCTL] -> ConfigFS
> conversion purposes, being able to call my do_configfs_[mkdir,rmdir]
> wrappers when existing legacy target functionality get called to
> add/remove the ConfigFS entries may have some value as well..  At least
> to make it easier for kernel level $FABRIC_MOD maintainers to see how
> generic kernel target mode configfs works... Just wanted to mention
> that..
> 
> >   I'm not clear if there can be more than one $FABRIC at a time.
> 
> Absoulutely.  Current each $FABRIC_MOD that registers itself with the
> common target engine calls the target_fabric_configfs_register() that
> creates its own struct config_group with fabric dependent configfs
> ops/abstractions.  This could very well be hidden behind the actual non
> configfs related target engine infrastructure when $FABRIC_MOD does
> actual fabric registration.  Also having common code for $FABRIC_MOD to
> use to allow those LUN mappings across multiple independent $FABRIC_MODs
> from target/core/$STORAGE_OBJECT is the main idea, but point taken that
> $FABRIC does not necessary need to be hanging directly off
> $CONFIGS/target to function..
> 
> Also why I think having $FABRIC under $CONFIGFS/target/ makes sense, I
> would like to be able to do rm -rf $CONFIGFS/target to shutdown all LUN
> mappings on all fabrics on all storage objects instead of rm -rf
> $CONFIGFS/iscsi ; rm -rf $CONFIGFS/sas ; rm -rf $CONFIGFS/pscsi ; rm -rf
> $CONFIGFS/target.. :-)
> 
> > etc.  That's what I'm trying to understand so I can help you out.
> > 
> 
> :-)
> 
> So, as of this morning I have basic iqn.foo/tpgt_1 ConfigFS
> functionality for LIO-Target.. Here is what it looks like..
> 
> <snip>
> 
> # Create some 
> 
> target:/sys/kernel/config/target/iscsi# mkdir -p iqn.iscsi-hd.renderbox/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p iqn.lio-production/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p iqn.upstreamtargetmodestorage/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p iqn.superturbodiskarray/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p `iscsi-name`/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p `iscsi-name`/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p `iscsi-name`/tpgt_1
> target:/sys/kernel/config/target/iscsi# mkdir -p `iscsi-name`/tpgt_1
> target:/sys/kernel/config/target/iscsi# tree
> .
> |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.11c045e90b
> |   `-- tpgt_1
> |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.acc810f5a2fc
> |   `-- tpgt_1
> |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.b341232f3595
> |   `-- tpgt_1
> |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.b5ada4ff123a
> |   `-- tpgt_1
> |-- iqn.iscsi-hd.renderbox
> |   `-- tpgt_1
> |-- iqn.lio-production
> |   `-- tpgt_1
> |-- iqn.superturbodiskarray
> |   `-- tpgt_1
> |-- iqn.upstreamtargetmodestorage
> |   `-- tpgt_1
> `-- lio_version
> 
> 16 directories, 1 file
> 
> # Notice the module count..
> target:/sys/kernel/config/target/iscsi# lsmod
> Module                  Size  Used by
> iscsi_target_mod      369004  17 
> target_core_configfs    11416  18 iscsi_target_mod
> configfs               24216  3 iscsi_target_mod,target_core_configfs
> 
> # Delete the IQNs and TPGTs created with /sbin/iscsi-name prefix.. 
> target:/sys/kernel/config/target/iscsi# rm -rf iqn.2003-01.org.linux-iscsi.target.i686\:sn.*
> 
> target:/sys/kernel/config/target/iscsi# tree
> .
> |-- iqn.iscsi-hd.renderbox
> |   `-- tpgt_1
> |-- iqn.lio-production
> |   `-- tpgt_1
> |-- iqn.superturbodiskarray
> |   `-- tpgt_1
> |-- iqn.upstreamtargetmodestorage
> |   `-- tpgt_1
> `-- lio_version
> 
> 8 directories, 1 file
> 
> # And the module counts..
> target:/sys/kernel/config/target/iscsi# lsmod
> Module                  Size  Used by
> iscsi_target_mod      369004  9 
> target_core_configfs    11416  10 iscsi_target_mod
> configfs               24216  3 iscsi_target_mod,target_core_configfs
> 

Ok, just commited first working code for ops for targetname, target
portal group and Network Portal <-> target portal group logic under
$CONFIGFS/target/iscsi/$IQN/

Here is what it looks like:

export CONFIGFS=/sys/kernel/config/
export FABRIC=/sys/kernel/config/target/iscsi

# Create four iSCSI Target Nodes with Target Portal Group Tag: 1
target:/# mkdir -p $FABRIC/`iscsi-name`/tpgt_1
target:/# mkdir -p $FABRIC/`iscsi-name`/tpgt_1
target:/# mkdir -p $FABRIC/`iscsi-name`/tpgt_1
target:/# mkdir -p $FABRIC/`iscsi-name`/tpgt_1
target:/# cd $CONFIGFS
target:/sys/kernel/config# tree
.
`-- target
    |-- iscsi
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.4b471a663865
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.60a44da1dea
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.7c988087f45c
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.ed63ac2a3da7
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   `-- lio_version
    `-- version
18 directories, 2 files

# Create some iSCSI network portals on the first two targetname+tpgt tuples
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.4b471a663865/tpgt_1/np/172.16.201.137:3260
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.4b471a663865/tpgt_1/np/172.16.201.137:3261
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.4b471a663865/tpgt_1/np/172.16.201.137:3262
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.4b471a663865/tpgt_1/np/172.16.201.137:3263
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.60a44da1dea/tpgt_1/np/172.16.201.137:3260
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.60a44da1dea/tpgt_1/np/172.16.201.137:3261
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.60a44da1dea/tpgt_1/np/172.16.201.137:3262
target:/sys/kernel/config# mkdir target/iscsi/iqn.2003-01.org.linux-iscsi.target.i686\:sn.60a44da1dea/tpgt_1/np/172.16.201.137:3263

target:/sys/kernel/config# tree
.
`-- target
    |-- iscsi
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.4b471a663865
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |           |-- 172.16.201.137:3260
    |   |           |   `-- portal_info
    |   |           |-- 172.16.201.137:3261
    |   |           |   `-- portal_info
    |   |           |-- 172.16.201.137:3262
    |   |           |   `-- portal_info
    |   |           `-- 172.16.201.137:3263
    |   |               `-- portal_info
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.60a44da1dea
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |           |-- 172.16.201.137:3260
    |   |           |   `-- portal_info
    |   |           |-- 172.16.201.137:3261
    |   |           |   `-- portal_info
    |   |           |-- 172.16.201.137:3262
    |   |           |   `-- portal_info
    |   |           `-- 172.16.201.137:3263
    |   |               `-- portal_info
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.7c988087f45c
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   |-- iqn.2003-01.org.linux-iscsi.target.i686:sn.ed63ac2a3da7
    |   |   `-- tpgt_1
    |   |       |-- lun
    |   |       `-- np
    |   `-- lio_version
    `-- version

26 directories, 10 files

Here is the commit:

http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commit;h=4098f1bcbe8406b882c4e265427f54a35f3c8b40

--nab