Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

[Josh Triplett]

On Wed, May 04, 2016 at 01:21:46AM +0200, Djalal Harouni wrote:
> This RFC tries to explore how to support filesystem operations inside
> user namespace using only VFS and a per mount namespace solution. This
> allows to take advantage of user namespace separations without
> introducing any change at the filesystems level. All this is handled
> with the virtual view of mount namespaces.
> 
> 1) Presentation:
> ================
> 
> The main aim is to support portable root filesystems and allow containers,
> virtual machines and other cases to use the same root filesystem.
> Due to security reasons, filesystems can't be mounted inside user
> namespaces, and mounting them outside will not solve the problem since
> they will show up with the wrong UIDs/GIDs. Read and write operations
> will also fail and so on.
[...]
> Using the user namespace in the super block seems the way to go, and
> there is the "Support fuse mounts in user namespaces" [1] patches which
> seem nice but perhaps too complex!? there is also the overlayfs solution,
> and finaly the VFS layer solution.
> 
> 
> We present here a simple VFS solution, everything is packed inside VFS,
> filesystems don't need to know anything (except probably XFS, and special
> operations inside union filesystems). Currently it supports ext4, btrfs
> and overlayfs. Changes into filesystems are small, just parse the
> vfs_shift_uids and vfs_shift_gids options during mount and set the
> appropriate flags into the super_block structure.

Interesting idea, and I certainly like the approach of addressing this
by mapping UIDs/GIDs within VFS.  However, I see a few issues with this:

> 2) The solution is based on VFS and mount namespaces, we use the user
> namespace of the containing mount namespace to check if we should shift
> UIDs/GIDs from/to virtual <=> on-disk view.
> If a filesystem was mounted with "vfs_shift_uids" and "vfs_shift_gids"
> options, and if it shows up inside a mount namespace that supports VFS
> UIDs/GIDs shifts then during each access we will remap UID/GID either
> to virtual or to on-disk view using simple helper functions to allow the
> access. In case the mount or current mount namespace do not support VFS
> UID/GID shifts, we fallback to the old behaviour, no shift is performed.
> 
> 3) inodes will always keep their original values which reflect the
> mapping inside init_user_ns which we consider the on-disk mapping.
> Therfore they will have a mapping from 0:65536 on-disk, these values are
> the persistent values that we have to write to the disk. We don't keep
> track of any UID/GID shift that was applied before. This gives
> portability and allows to use the previous mapping which was freed for
> another root filesystem...

What about filesystems that support 32-bit UIDs/GIDs on disk, which
includes most modern filesystems, including ext4?

What about nesting, which seems like a perfectly legitimate thing to do?
(For instance, you want to use a container to run the equivalent of a
distribution-in-a-chroot, and that distribution internally uses
containers/namespaces for its own purposes, such as running daemons with
lower privileges.)  The "shift" approach can't support that, because you
can't give the namespace in the middle permission to "shift" to
UIDs/GIDs it doesn't control.

I do very much like the idea of remapping UIDs/GIDs within VFS.
However, I'm wondering if it would work better to provide a
uidmap/gidmap, similar to that used for userns today.  In simple cases,
that should be approximately as efficient as the approach in this patch
series (if it maps a range of UIDs inside to UIDs outside, it would
effectively be a bounds-check, an add, and a fallback value for unmapped
IDs).  But you could then nest it the same way you can nest uidmap: as
root in a namespace, you can map any UID you yourself were given.

This would also be useful for non-container applications: for instance,
you could mount a USB disk with an ext4 filesystem and not assume the
UIDs match those of your host system, while also not squashing them all
to a single UID the way a uid= option would.

As with this series, a mapping approach wouldn't require allowing mounts
inside the namespace.  You *could* mount from within the namespace if
you want and you have appropriate access to do so, mapping UIDs/GIDs on
the disk to UIDs/GIDs you have in your namespace.  However, you could
also do the mount with mapping from outside the namespace, to non-root
UIDs/GIDs, and then use those same UIDs/GIDs in a userns mapping to map
them to root.

The main design constraint with a full mapping would be passing that
through "mount".  There have been discussions on and off for years about
replacing the mount() system call with something either two-phase (get
filesystem driver FD, send it a series of parameters ending with mount;
the VFS would interpret many of those parameters) or three-phase (get
filesystem driver FD, send it parameters ending with getting a directory
FD, bind the directory FD).  Given an interface like that, providing a
UID/GID map at mount time seems plausible.

Alternatively, a much simpler approach that could potentially be
expanded in the future would be to add *two* parameters each for UID and
GID: a base and a max.  That would define a range, which doesn't
necessarily need to be exactly 2**16; thus, if you had a big enough
range, that approach would nest as well.

- Josh Triplett

Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

[Djalal Harouni]

Hi Josh,

Thanks for the reply! I'll resend this RFC soon, as it seems
that I receive from mailing lists but can't send...
probably some filters... my domain was dead for a short period...

On Tue, May 03, 2016 at 05:41:07PM -0700, Josh Triplett wrote:
> On Wed, May 04, 2016 at 01:21:46AM +0200, Djalal Harouni wrote:
> > This RFC tries to explore how to support filesystem operations inside
> > user namespace using only VFS and a per mount namespace solution. This
> > allows to take advantage of user namespace separations without
> > introducing any change at the filesystems level. All this is handled
> > with the virtual view of mount namespaces.
> > 
> > 1) Presentation:
> > ================
> > 
> > The main aim is to support portable root filesystems and allow containers,
> > virtual machines and other cases to use the same root filesystem.
> > Due to security reasons, filesystems can't be mounted inside user
> > namespaces, and mounting them outside will not solve the problem since
> > they will show up with the wrong UIDs/GIDs. Read and write operations
> > will also fail and so on.
> [...]
> > Using the user namespace in the super block seems the way to go, and
> > there is the "Support fuse mounts in user namespaces" [1] patches which
> > seem nice but perhaps too complex!? there is also the overlayfs solution,
> > and finaly the VFS layer solution.
> > 
> > 
> > We present here a simple VFS solution, everything is packed inside VFS,
> > filesystems don't need to know anything (except probably XFS, and special
> > operations inside union filesystems). Currently it supports ext4, btrfs
> > and overlayfs. Changes into filesystems are small, just parse the
> > vfs_shift_uids and vfs_shift_gids options during mount and set the
> > appropriate flags into the super_block structure.
> 
> Interesting idea, and I certainly like the approach of addressing this
> by mapping UIDs/GIDs within VFS.  However, I see a few issues with this:
> 
> > 2) The solution is based on VFS and mount namespaces, we use the user
> > namespace of the containing mount namespace to check if we should shift
> > UIDs/GIDs from/to virtual <=> on-disk view.
> > If a filesystem was mounted with "vfs_shift_uids" and "vfs_shift_gids"
> > options, and if it shows up inside a mount namespace that supports VFS
> > UIDs/GIDs shifts then during each access we will remap UID/GID either
> > to virtual or to on-disk view using simple helper functions to allow the
> > access. In case the mount or current mount namespace do not support VFS
> > UID/GID shifts, we fallback to the old behaviour, no shift is performed.
> > 
> > 3) inodes will always keep their original values which reflect the
> > mapping inside init_user_ns which we consider the on-disk mapping.
> > Therfore they will have a mapping from 0:65536 on-disk, these values are
> > the persistent values that we have to write to the disk. We don't keep
> > track of any UID/GID shift that was applied before. This gives
> > portability and allows to use the previous mapping which was freed for
> > another root filesystem...
> 
> What about filesystems that support 32-bit UIDs/GIDs on disk, which
> includes most modern filesystems, including ext4?
That 0:65536 was just an example, we are not hardcoding anything here, we
just use the mapping provided by user namespaces which supports 32bits.

That plan was just an example where containers will setup the lower
16bits for UID/GID shifts inside the container. This way, they are able
to have the same "virtual" mapping inside the container and to use the
rootfs, where outside they could use the upper 16bits for separation or
as a container ID, or whatever.


> What about nesting, which seems like a perfectly legitimate thing to do?
> (For instance, you want to use a container to run the equivalent of a
> distribution-in-a-chroot, and that distribution internally uses
> containers/namespaces for its own purposes, such as running daemons with
> lower privileges.)  The "shift" approach can't support that, because you
> can't give the namespace in the middle permission to "shift" to
> UIDs/GIDs it doesn't control.
I'll confirm this later and respond.


> I do very much like the idea of remapping UIDs/GIDs within VFS.
> However, I'm wondering if it would work better to provide a
> uidmap/gidmap, similar to that used for userns today.  In simple cases,
> that should be approximately as efficient as the approach in this patch
> series (if it maps a range of UIDs inside to UIDs outside, it would
> effectively be a bounds-check, an add, and a fallback value for unmapped
> IDs).  But you could then nest it the same way you can nest uidmap: as
> root in a namespace, you can map any UID you yourself were given.
> 
> This would also be useful for non-container applications: for instance,
> you could mount a USB disk with an ext4 filesystem and not assume the
> UIDs match those of your host system, while also not squashing them all
> to a single UID the way a uid= option would.
> 
> As with this series, a mapping approach wouldn't require allowing mounts
> inside the namespace.  You *could* mount from within the namespace if
> you want and you have appropriate access to do so, mapping UIDs/GIDs on
> the disk to UIDs/GIDs you have in your namespace.  However, you could
> also do the mount with mapping from outside the namespace, to non-root
> UIDs/GIDs, and then use those same UIDs/GIDs in a userns mapping to map
> them to root.
> 
> The main design constraint with a full mapping would be passing that
> through "mount".  There have been discussions on and off for years about
> replacing the mount() system call with something either two-phase (get
> filesystem driver FD, send it a series of parameters ending with mount;
> the VFS would interpret many of those parameters) or three-phase (get
> filesystem driver FD, send it parameters ending with getting a directory
> FD, bind the directory FD).  Given an interface like that, providing a
> UID/GID map at mount time seems plausible.
Could you please provide some links for these discussions ?

I'll get back to it.

> Alternatively, a much simpler approach that could potentially be
> expanded in the future would be to add *two* parameters each for UID and
> GID: a base and a max.  That would define a range, which doesn't
> necessarily need to be exactly 2**16; thus, if you had a big enough
> range, that approach would nest as well.
Hm, I can see but I'm not sure if it will make sense, since this
will hardcode the mapping during mount ? where maybe that mount can be
used later for another mapping configuration ? I think we should just
get a user namespace reference and that's it. Now we just allow the
current user namespace interface to do the job for us, and as said above
the 2**16 is just an example.

> 
> - Josh Triplett

Thanks!

-- 
Djalal Harouni
http://opendz.org

Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

[Josh Triplett]

On Wed, May 04, 2016 at 11:08:42AM +0100, Djalal Harouni wrote:
> On Tue, May 03, 2016 at 05:41:07PM -0700, Josh Triplett wrote:
> > The main design constraint with a full mapping would be passing that
> > through "mount".  There have been discussions on and off for years about
> > replacing the mount() system call with something either two-phase (get
> > filesystem driver FD, send it a series of parameters ending with mount;
> > the VFS would interpret many of those parameters) or three-phase (get
> > filesystem driver FD, send it parameters ending with getting a directory
> > FD, bind the directory FD).  Given an interface like that, providing a
> > UID/GID map at mount time seems plausible.
> Could you please provide some links for these discussions ?
> 
> I'll get back to it.

I don't know of a good example of those discussions occurring in public;
they've occurred at Kernel Summit for years.

Al Viro would know if they've been discusssed publically.  Al?

> > Alternatively, a much simpler approach that could potentially be
> > expanded in the future would be to add *two* parameters each for UID and
> > GID: a base and a max.  That would define a range, which doesn't
> > necessarily need to be exactly 2**16; thus, if you had a big enough
> > range, that approach would nest as well.
> Hm, I can see but I'm not sure if it will make sense, since this
> will hardcode the mapping during mount ? where maybe that mount can be
> used later for another mapping configuration ? I think we should just
> get a user namespace reference and that's it. Now we just allow the
> current user namespace interface to do the job for us, and as said above
> the 2**16 is just an example.

Please ignore this last paragraph; it was based on my misunderstanding
the approach you took.

- Josh Triplett

Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

[Seth Forshee]

On Wed, May 04, 2016 at 01:21:46AM +0200, Djalal Harouni wrote:
> This RFC tries to explore how to support filesystem operations inside
> user namespace using only VFS and a per mount namespace solution. This
> allows to take advantage of user namespace separations without
> introducing any change at the filesystems level. All this is handled
> with the virtual view of mount namespaces.
> 
> 
> 1) Presentation:
> ================
> 
> The main aim is to support portable root filesystems and allow containers,
> virtual machines and other cases to use the same root filesystem.
> Due to security reasons, filesystems can't be mounted inside user
> namespaces, and mounting them outside will not solve the problem since
> they will show up with the wrong UIDs/GIDs. Read and write operations
> will also fail and so on.
> 
> The current userspace solution is to automatically chown the whole root
> filesystem before starting a container, example:
> (host) init_user_ns  1000000:1065536  => (container) user_ns_X1 0:65535
> (host) init_user_ns  2000000:2065536  => (container) user_ns_Y1 0:65535
> (host) init_user_ns  3000000:3065536  => (container) user_ns_Z1 0:65535
> ...
> 
> Every time a chown is called, files are changed and so on... This
> prevents to have portable filesystems where you can throw anywhere
> and boot. Having an extra step to adapt the filesystem to the current
> mapping and persist it will not allow to verify its integrity, it makes
> snapshots and migration a bit harder, and probably other limitations...
> 
> It seems that there are multiple ways to allow user namespaces combine
> nicely with filesystems, but none of them is that easy. The bind mount
> and pin the user namespace during mount time will not work, bind mounts
> share the same super block, hence you may endup working on the wrong
> vfsmount context and there is no easy way to get out of that...
> 
> Using the user namespace in the super block seems the way to go, and
> there is the "Support fuse mounts in user namespaces" [1] patches which
> seem nice but perhaps too complex!? there is also the overlayfs solution,
> and finaly the VFS layer solution.

I'm not sure if you're meaning to propose your patches as an alternative
to mine or not, but I think they're orthogonal. My goal is to allow
containers in user namespaces to mount some subset of filesystem types
(not specifically container root filesystems, but in general), which
your patches won't enable. Your goal is to share a rootfs between
multiple containers with different uid/gid shifts, which my patches
don't help with.

> We present here a simple VFS solution, everything is packed inside VFS,
> filesystems don't need to know anything (except probably XFS, and special
> operations inside union filesystems). Currently it supports ext4, btrfs
> and overlayfs. Changes into filesystems are small, just parse the
> vfs_shift_uids and vfs_shift_gids options during mount and set the
> appropriate flags into the super_block structure.
> 
> 1) Filesystems don't need the FS_USERNS_MOUNT flag, so no user
> namespace mounting, they stay secure, nothing changes.
> 
> 2) The solution is based on VFS and mount namespaces, we use the user
> namespace of the containing mount namespace to check if we should shift
> UIDs/GIDs from/to virtual <=> on-disk view.
> If a filesystem was mounted with "vfs_shift_uids" and "vfs_shift_gids"
> options, and if it shows up inside a mount namespace that supports VFS
> UIDs/GIDs shifts then during each access we will remap UID/GID either
> to virtual or to on-disk view using simple helper functions to allow the
> access. In case the mount or current mount namespace do not support VFS
> UID/GID shifts, we fallback to the old behaviour, no shift is performed.
> 
> 3) inodes will always keep their original values which reflect the
> mapping inside init_user_ns which we consider the on-disk mapping.
> Therfore they will have a mapping from 0:65536 on-disk, these values are
> the persistent values that we have to write to the disk. We don't keep
> track of any UID/GID shift that was applied before. This gives
> portability and allows to use the previous mapping which was freed for
> another root filesystem...

Sorry, I haven't had time to look at the patches, but how are you
handling suid/sgid? Will the process get the ids in the inode or the
shifted ids?

Thanks,
Seth

Re: [RFC PATCH 0/0] VFS:userns: support portable root filesystems

[Djalal Harouni]

Hi,

On Wed, May 04, 2016 at 08:34:04AM -0500, Seth Forshee wrote:
> On Wed, May 04, 2016 at 01:21:46AM +0200, Djalal Harouni wrote:
> > This RFC tries to explore how to support filesystem operations inside
> > user namespace using only VFS and a per mount namespace solution. This
> > allows to take advantage of user namespace separations without
> > introducing any change at the filesystems level. All this is handled
> > with the virtual view of mount namespaces.
> > 
> > 
> > 1) Presentation:
> > ================
> > 
> > The main aim is to support portable root filesystems and allow containers,
> > virtual machines and other cases to use the same root filesystem.
> > Due to security reasons, filesystems can't be mounted inside user
> > namespaces, and mounting them outside will not solve the problem since
> > they will show up with the wrong UIDs/GIDs. Read and write operations
> > will also fail and so on.
> > 
> > The current userspace solution is to automatically chown the whole root
> > filesystem before starting a container, example:
> > (host) init_user_ns  1000000:1065536  => (container) user_ns_X1 0:65535
> > (host) init_user_ns  2000000:2065536  => (container) user_ns_Y1 0:65535
> > (host) init_user_ns  3000000:3065536  => (container) user_ns_Z1 0:65535
> > ...
> > 
> > Every time a chown is called, files are changed and so on... This
> > prevents to have portable filesystems where you can throw anywhere
> > and boot. Having an extra step to adapt the filesystem to the current
> > mapping and persist it will not allow to verify its integrity, it makes
> > snapshots and migration a bit harder, and probably other limitations...
> > 
> > It seems that there are multiple ways to allow user namespaces combine
> > nicely with filesystems, but none of them is that easy. The bind mount
> > and pin the user namespace during mount time will not work, bind mounts
> > share the same super block, hence you may endup working on the wrong
> > vfsmount context and there is no easy way to get out of that...
> > 
> > Using the user namespace in the super block seems the way to go, and
> > there is the "Support fuse mounts in user namespaces" [1] patches which
> > seem nice but perhaps too complex!? there is also the overlayfs solution,
> > and finaly the VFS layer solution.
> 
> I'm not sure if you're meaning to propose your patches as an alternative
> to mine or not, but I think they're orthogonal. My goal is to allow
> containers in user namespaces to mount some subset of filesystem types

I see. Yes I didn't think that much about mounting filesystems inside
user namespaces, and that's not the mainly use case here for sure.

> (not specifically container root filesystems, but in general), which
> your patches won't enable. Your goal is to share a rootfs between
> multiple containers with different uid/gid shifts, which my patches
> don't help with.
The aim with these patches is containers but not only that.

Basically it goes to every situations where you need user namespaces
for some separation but UIDs/GIDs will end up with the wrong identity.
So yes sharing rootfs but also to give the correct UIDs/GIDs mapping
inside user namespaces.

Maybe both can be combined ? I checked your series lately but to be
honest I'm not that familiar with the use case. This one is just an RFC
for the moment, and ultimatly it's designed to stay *deep* inside *VFS*.
Nothing is exposed even the flags inside the super_block are internal
flags, the VFS should do the work for us, and maybe XFS and rare cases
will have to call VFS helpers but that's it. So I tried as much as
possible to let the thing flexible, and at the same time hide the
details and use existing interfaes, so yes maybe something can be worked
out ?


> > We present here a simple VFS solution, everything is packed inside VFS,
> > filesystems don't need to know anything (except probably XFS, and special
> > operations inside union filesystems). Currently it supports ext4, btrfs
> > and overlayfs. Changes into filesystems are small, just parse the
> > vfs_shift_uids and vfs_shift_gids options during mount and set the
> > appropriate flags into the super_block structure.
> > 
> > 1) Filesystems don't need the FS_USERNS_MOUNT flag, so no user
> > namespace mounting, they stay secure, nothing changes.
> > 
> > 2) The solution is based on VFS and mount namespaces, we use the user
> > namespace of the containing mount namespace to check if we should shift
> > UIDs/GIDs from/to virtual <=> on-disk view.
> > If a filesystem was mounted with "vfs_shift_uids" and "vfs_shift_gids"
> > options, and if it shows up inside a mount namespace that supports VFS
> > UIDs/GIDs shifts then during each access we will remap UID/GID either
> > to virtual or to on-disk view using simple helper functions to allow the
> > access. In case the mount or current mount namespace do not support VFS
> > UID/GID shifts, we fallback to the old behaviour, no shift is performed.
> > 
> > 3) inodes will always keep their original values which reflect the
> > mapping inside init_user_ns which we consider the on-disk mapping.
> > Therfore they will have a mapping from 0:65536 on-disk, these values are
> > the persistent values that we have to write to the disk. We don't keep
> > track of any UID/GID shift that was applied before. This gives
> > portability and allows to use the previous mapping which was freed for
> > another root filesystem...
> 
> Sorry, I haven't had time to look at the patches, but how are you
> handling suid/sgid? Will the process get the ids in the inode or the
> shifted ids?
Hmm, what do you mean by this please ? 

I picked a patch from you series, the one that was written by Andy and
adapt it here so outsiders of the mount namespace and its user namespace
can't fool us. Please see patch 0003, but I'm sure that it needs a
better review as it is now just an RFC.

Thank you!


> Thanks,
> Seth
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-security-module" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Djalal Harouni
http://opendz.org