[GIT *] make headers_install

[GIT *] make headers_install

[David Woodhouse]

Linus, please pull from git://git.infradead.org/hdrinstall-2.6.git

This contains an implementation of a 'make headers_install' target for
the kernel -- based on original work by Arnd Bergmann, modifed my myself
and then cleaned up by Sam. This copies _selected_ kernel headers out to
a separate directory, passing them through sed and BSD's 'unifdef' tool
to remove parts which userspace should not see.

(The BSD 'unifdef' tool is available at least in Fedora and Debian
through their standard package management tools. I believe that Sam
intends to follow up with a patch to add our own copy of unifdef into
the kernel scripts/ directory, as soon as he's fixed the dependency
issues with that.)

This isn't a departure from our current policy that random userspace
must not poke at kernel private headers. It's just an attempt to impose
some control over those places where we have to accept that people _do_
use the kernel's headers -- when building system libraries and tools,
and when building compilers.

Currently, the compiler-build scripts just use 'cp -a', while the
distributions tend to do their own thing to make it _slightly_ saner
than that, although it's a lot of work to do so. The result is wildly
inconsistent and often exposes things which we really don't want
userspace to have copies of.

By adding a 'make headers_install' target to the kernel, we regulate
those people who really do have to use kernel headers, and we can ensure
that we have a _consistent_ set of headers across all distributions,
which contains only what we _need_ to expose; ioctl definitions, etc.

An additional benefit is that comparing the results of 'make
headers_install' from one kernel release to the next allows us to spot
kernel<->user ABI changes in isolation and give them the extra review
that they deserve. I've already caught and fixed one potential problem
with 32-bit userspace on a 64-bit kernel this way.

The result of this export is already being used in the Fedora Core 6
test releases, and other distributions are either looking at switching
over to it or have done so already.

Ignoring the new Kbuild files which just list the files to be exported
from each directory under include/, the diffstat is as follows:

 Makefile                              |   17 ++++
 scripts/Makefile.headersinst          |  158 +++++++++++++++++++++++++++++++++
 scripts/hdrcheck.sh                   |    8 ++
 49 files changed, 399 insertions(+), 0 deletions(-)

David S. Miller:
      Restrict headers exported to userspace for SPARC and SPARC64

David Woodhouse:
      Basic implementation of 'make headers_install'
      Basic implementation of 'make headers_check'
      Add generic Kbuild files for 'make headers_install'
      Add Kbuild file for PowerPC 'make headers_install'
      Add Kbuild file for x86_64 'make headers_install'
      Add Kbuild file for i386 'make headers_install'
      Add Kbuild file for S390 'make headers_install'
      Add Kbuild file for IA64 'make headers_install'
      Add Kbuild file for SPARC 'make headers_install'
      Add Kbuild file for Alpha 'make headers_install'
      Add empty Kbuild files for 'make headers_install' in remaining arches.

Jean Delvare:
      Remove <linux/i2c-id.h> and <linux/i2c-algo-ite.h> from userspace export

-- 
dwmw2

Re: [GIT *] make headers_install

[Ralf Baechle]

On Tue, Jun 27, 2006 at 11:12:52PM +0100, David Woodhouse wrote:

> Linus, please pull from git://git.infradead.org/hdrinstall-2.6.git
> 
> This contains an implementation of a 'make headers_install' target for
> the kernel -- based on original work by Arnd Bergmann, modifed my myself
> and then cleaned up by Sam. This copies _selected_ kernel headers out to
> a separate directory, passing them through sed and BSD's 'unifdef' tool
> to remove parts which userspace should not see.
> 
> (The BSD 'unifdef' tool is available at least in Fedora and Debian
> through their standard package management tools. I believe that Sam
> intends to follow up with a patch to add our own copy of unifdef into
> the kernel scripts/ directory, as soon as he's fixed the dependency
> issues with that.)
> 
> This isn't a departure from our current policy that random userspace
> must not poke at kernel private headers. It's just an attempt to impose
> some control over those places where we have to accept that people _do_
> use the kernel's headers -- when building system libraries and tools,
> and when building compilers.
> 
> Currently, the compiler-build scripts just use 'cp -a', while the
> distributions tend to do their own thing to make it _slightly_ saner
> than that, although it's a lot of work to do so. The result is wildly
> inconsistent and often exposes things which we really don't want
> userspace to have copies of.
> 
> By adding a 'make headers_install' target to the kernel, we regulate
> those people who really do have to use kernel headers, and we can ensure
> that we have a _consistent_ set of headers across all distributions,
> which contains only what we _need_ to expose; ioctl definitions, etc.
> 
> An additional benefit is that comparing the results of 'make
> headers_install' from one kernel release to the next allows us to spot
> kernel<->user ABI changes in isolation and give them the extra review
> that they deserve. I've already caught and fixed one potential problem
> with 32-bit userspace on a 64-bit kernel this way.
> 
> The result of this export is already being used in the Fedora Core 6
> test releases, and other distributions are either looking at switching
> over to it or have done so already.
> 
> Ignoring the new Kbuild files which just list the files to be exported
> from each directory under include/, the diffstat is as follows:

Thanks for the time you've been pouring into this.  Copying kernel header
into applications clearly didn't work too well, especially for arch stuff
(two dozen and counting ...) it was a pain and each distribution had
different sets of hacked kernel headers.  So I hope this will restore
some sanity - and same for klibc.

  Ralf

Re: [GIT *] make headers_install

[David Woodhouse]

On Thu, 2006-06-29 at 00:44 +0100, Ralf Baechle wrote:
> Thanks for the time you've been pouring into this.  Copying kernel
> header into applications clearly didn't work too well, especially for
> arch stuff (two dozen and counting ...) it was a pain and each
> distribution had different sets of hacked kernel headers.

The technical part wasn't very time-consuming at all. Arnd provided the
basic implementation, and I just tweaked it a little to use unifdef and
be a bit more selective about what we export. It's probably taken less
time than it would have done to get Fedora's 'glibc-kernheaders' package
into shape manually the way it always used to be done -- and it'll
_certainly_ be a Godsend for future updates.

The time-consuming part was chasing up those who look after similar
packages in other distributions and making sure they were happy enough
with the principle too -- tracking them down on IRC when they ignored my
emails.

The current hurdle seems to be getting Linus to take it or at least
comment, now that everyone _else_ seems to be fairly much in agreement.
I'm hoping we don't have to let it drag on till the Kernel Summit.

-- 
dwmw2

[GIT *] make headers_install

[David Woodhouse]

Linus, please pull from git://git.infradead.org/hdrinstall-2.6.git 

This implements a 'headers_install' make target, which takes a selected
subset of kernel headers and exports them in a form which is usable by
system libraries and tools, also removing all #ifdef __KERNEL__ from
them.

This provides a number of benefits to all concerned.

 - it makes life a _lot_ easier for those who have to maintain such a
   set of headers for distributions, or for building toolchains.
   Previously, these were often maintained manually, adding new
   structures and ioctls to a set of header files which were originally
   copied from an older kernel and cleaned up. Now, they can be obtained
   directly in a form which is useful. It was the fact that I own the
   'glibc-kernheaders' package in Fedora which finally prompted me to
   follow up on this.

 - it makes life easier for those whose libraries or tools must _use_
   such headers, because it means that the set of available headers can
   be _consistent_ across all distributions and toolchains rather than
   having random differences.

 - that consistency also allows us to reduce the abuse of kernel headers
   by random userspace, by removing files which userspace has no
   business with. For example, we can simply drop asm/atomic.h from the
   export to stop people from abusing it in userspace -- when we did
   that kind of thing in Fedora alone, we got lots of complaints that
   "it works in $OTHERDISTRO". (I haven't done that yet except in the
   Fedora version; I was concentrating on the mechanism before we start
   to enforce such policy).

 - we can take diffs between the exported headers from one version to
   the next, and see any ABI-affecting changes clearly. This should help
   us to keep tabs on ABI changes. I've already kicked the GFS2 guys and
   had them fix a problem with 32-bit userspace on 64-bit kernel,
   because after 'make headers_install' I was able to see their ABI
   change in isolation instead of hidden amongst the rest of their
   patches.

 - The additional 'make headers_check' target also allows us to run some
   basic sanity checks on the exported headers. Currently, I've only
   checked that they don't attempt to include headers which aren't
   marked for export -- but Arjan van de Ven has offered patches which
   go a little further than that. We could check for bogus types like
   'u32' being used instead of '__u32', etc. And perhaps we could even 
   check that headers are _compilable_ in their standalone form,
   although that isn't usually assumed to be the case since they don't
   all include all their dependencies. Still, the mechanism is there for
   us to do with it as we see fit.

It's based on an original implementation by Arnd Bergmann, hacked around
a bit by myself, and reviewed and cleaned up by Sam Ravnborg. It
includes some additional processing provided by Tim Yamin who handles
kernel headers for the Gentoo distribution.

It uses the BSD 'unifdef' tool, which is available by 'yum install
unifdef' on Fedora systems, and I believe is also available by similar
means in Debian. Sam's actually expressed a desire to stick a copy of
unifdef.c into the kernel's scripts/ directory, and Tony Finch (the BSD
author/maintainer) has agreed to that -- Sam will probably follow up
with an appropriate patch shortly, once he's sorted out the dependency
issues with that and cross-building.

The result of this export is already shipping in the 'glibc-kernheaders'
package in Fedora Core 6 test 1. I've also been working with those who own
equivalent packages in other distributions, and there is a _lot_ of
interest in switching over to this method.

Ignoring the new Kbuild files which just list the files to be exported
from each directory under include/, the diffstat is as follows:

 Makefile                              |   17 ++++
 scripts/Makefile.headersinst          |  158 +++++++++++++++++++++++++++++++++
 scripts/hdrcheck.sh                   |    8 ++
 49 files changed, 399 insertions(+), 0 deletions(-)

David S. Miller:
      Restrict headers exported to userspace for SPARC and SPARC64

David Woodhouse:
      Basic implementation of 'make headers_install'
      Basic implementation of 'make headers_check'
      Add generic Kbuild files for 'make headers_install'
      Add Kbuild file for PowerPC 'make headers_install'
      Add Kbuild file for x86_64 'make headers_install'
      Add Kbuild file for i386 'make headers_install'
      Add Kbuild file for S390 'make headers_install'
      Add Kbuild file for IA64 'make headers_install'
      Add Kbuild file for SPARC 'make headers_install'
      Add Kbuild file for Alpha 'make headers_install'
      Add empty Kbuild files for 'make headers_install' in remaining arches.
      Remove export of include/linux/isdn/tpam.h

Jean Delvare:
      Remove <linux/i2c-id.h> and <linux/i2c-algo-ite.h> from userspace export

-- 
dwmw2