[PATCH] MODULE_FIRMWARE for binary firmware(s)

[PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

From: Jon Masters <jcm@redhat.com>

Right now, various kernel modules are being migrated over to use
request_firmware in order to pull in binary firmware blobs from userland
when the module is loaded. This makes sense.

However, there is right now little mechanism in place to automatically
determine which binary firmware blobs must be included with a kernel in
order to satisfy the prerequisites of these drivers. This affects
vendors, but also regular users to a certain extent too.

The attached patch introduces MODULE_FIRMWARE as a mechanism for
advertising that a particular firmware file is to be loaded - it will
then show up via modinfo and could be used e.g. when packaging a kernel.

Signed-off-by: Jon Masters <jcm@redhat.com>

diff -urN linux-2.6.16.2_orig/include/linux/module.h linux-2.6.16.2_dev/include/linux/module.h
--- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07 17:56:47.000000000 +0100
+++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12 13:51:56.000000000 +0100
@@ -155,6 +155,8 @@
 */
 #define MODULE_VERSION(_version) MODULE_INFO(version, _version)

+#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
+
 /* Given an address, look for it in the exception tables */
 const struct exception_table_entry *search_exception_tables(unsigned long add);

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

Jon,
   would it be possible to have something less then an initrd that would 
allow the firmware blob to be packaged with the kernel? Your approach is 
just fine if the things that will need firmware are compiled as modules 
(the firmware can be placed on the same filesystem image as the modules 
that need it) but some people prefer to build monolithic kernel images, 
and in those cases there is no way for the kernel to get the firmware as 
there is no filesystem available yet.

I know that Rob Landley is doing some tricks with firmware linux to append 
an initramfs to the kernel image (http://www.landley.net/code/firmware/). 
There is already support in the kconfig for specifying the initramfs 
source, why culdn't we have the build process fetch any firmware needed by 
the non-modular portions and store them at the end of the kernel image so 
that they can be found during boot.

if nothing else make a skeleton initramfs that contains the modules and 
just enough other logic to continue the boot process as if the initramfs 
wasn't involved.

right now I have a laptop that's not working on wireless due to this exact 
problem (ipw2200 driver, and I haven't taken the time to setup modules for 
it yet)

David Lang


On Wed, 19 Apr 2006, Jon Masters wrote:

> Date: Wed, 19 Apr 2006 00:41:56 +0100
> From: Jon Masters <jonathan@jonmasters.org>
> To: akpm@osdl.org, linux-kernel@vger.kernel.org
> Subject: [PATCH] MODULE_FIRMWARE for binary firmware(s)
> 
> From: Jon Masters <jcm@redhat.com>
>
> Right now, various kernel modules are being migrated over to use
> request_firmware in order to pull in binary firmware blobs from userland
> when the module is loaded. This makes sense.
>
> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
>
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.
>
> Signed-off-by: Jon Masters <jcm@redhat.com>
>
> diff -urN linux-2.6.16.2_orig/include/linux/module.h linux-2.6.16.2_dev/include/linux/module.h
> --- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07 17:56:47.000000000 +0100
> +++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12 13:51:56.000000000 +0100
> @@ -155,6 +155,8 @@
> */
> #define MODULE_VERSION(_version) MODULE_INFO(version, _version)
>
> +#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
> +
> /* Given an address, look for it in the exception tables */
> const struct exception_table_entry *search_exception_tables(unsigned long add);
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, David Lang <dlang@digitalinsight.com> wrote:

>    would it be possible to have something less then an initrd that would
> allow the firmware blob to be packaged with the kernel?

With some modifications perhaps. I don't know if I see the value tbh :-)

> Your approach is just fine if the things that will need firmware are
> compiled as modules

Hmmm. Yeah. I'm not sure what the general feeling is on this - I'm
tempted to say that we expect modules to be used and that if they're
not then the vendor/user has to do the hoop jumping for themselves.
This code won't stop you from making a monolithic kernel and
satisfying any module requirements for yourself :-)

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[David Lang]

On Wed, 19 Apr 2006, Jon Masters wrote:

>> Your approach is just fine if the things that will need firmware are
>> compiled as modules
>
> Hmmm. Yeah. I'm not sure what the general feeling is on this - I'm
> tempted to say that we expect modules to be used and that if they're
> not then the vendor/user has to do the hoop jumping for themselves.
> This code won't stop you from making a monolithic kernel and
> satisfying any module requirements for yourself :-)

Two things with this.

1. there is no way to satisfy the firmware requirements currently

2. I thought I heard Linus state recently that makeing something only work 
as a module was unacceptable, officially stateing that modules are 
required and monolithic kernels aren't allowed anymore doesn't sound 
reasonable.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, David Lang <dlang@digitalinsight.com> wrote:

> 2. I thought I heard Linus state recently that makeing something only work
> as a module was unacceptable, officially stateing that modules are
> required and monolithic kernels aren't allowed anymore doesn't sound
> reasonable.

It works fine with or without modules - it's just that if you don't
have the module then you have to supply the firmware for yourself.
It's just like any of the other MODULE_ tags.

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Duncan Sands]

Hi Jon,

> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
> 
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.

I haven't really understood what problem this solves.  Is this just a
standardised form of documentation, or are you imagining that an automatic
tool will use this to auto include a minimal set of firmware files in an
initrd?  I'm thinking of something like this: (1) redhat (or whoever) ships
firmware files for every driver under the sun in /lib/firmware; (2) redhat
wants to allow users to have a customized initrd with only essential drivers;
(3) the tool goes through the list of essential drivers, looks up the firmware
string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
in the initrd.

All the best,

Duncan.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:

> Hi Jon,

Hi Duncan.

> > However, there is right now little mechanism in place to automatically
> > determine which binary firmware blobs must be included with a kernel in
> > order to satisfy the prerequisites of these drivers. This affects
> > vendors, but also regular users to a certain extent too.
> >
> > The attached patch introduces MODULE_FIRMWARE as a mechanism for
> > advertising that a particular firmware file is to be loaded - it will
> > then show up via modinfo and could be used e.g. when packaging a kernel.

> I haven't really understood what problem this solves.  Is this just a
> standardised form of documentation, or are you imagining that an automatic
> tool will use this to auto include a minimal set of firmware files in an
> initrd?

I'm imagining that the resultant modinfo output can be used by a tool
for anyone to package up the correct firmware to go with a given
driver. Right now, there's no way to do that - i.e. we've gone
backwards from a standpoint of coupling a kernel with firmware. I
completely understand why firmware doesn't really belong in the
kernel, so let's add this :-)

> I'm thinking of something like this: (1) redhat (or whoever) ships
> firmware files for every driver under the sun in /lib/firmware; (2) redhat
> wants to allow users to have a customized initrd with only essential drivers;
> (3) the tool goes through the list of essential drivers, looks up the firmware
> string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
> in the initrd.

That kind of thing. It's not just Red Hat who benefit - anyone who
wants to package up a kernel and do something with it will want to
know about firmware they might need. Including everything in
/lib/firmware "just in case" is as ugly as having userspace tools with
duplicated logic that need to understand about the internals of a
driver module.

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Duncan Sands]

> > I haven't really understood what problem this solves.  Is this just a
> > standardised form of documentation, or are you imagining that an automatic
> > tool will use this to auto include a minimal set of firmware files in an
> > initrd?
> 
> I'm imagining that the resultant modinfo output can be used by a tool
> for anyone to package up the correct firmware to go with a given
> driver.

If a tool is to do the packaging, then this means that the firmware must
already be present on the machine, for example in /lib/firmware.  Logically
speaking, that means the role of any tool is to select a subset of the files
in /lib/firmware, for example a minimal set for inclusion in an initrd.  After
all, it can't add new files that don't exist on the machine, since where would
it get them from?  (I suppose it could download them from "firmware central").
Is there any use for such a tool, besides creating small initrds?  How big an
issue is that?  Is there really any reason not to simply throw everything in
/lib/firmware into any initrd that is created?  But perhaps you are thinking of
the suppliers of a distribution who have a gazillion firmware files that they've
collected over the years, some of which are too old and some of which are too
recent for the drivers that will be shipped; and the tool is to select the subset
which is relevant for the shipped driver versions?  i.e. the firmware selection
process is not done on each end user's machine, but by the people preparing the
distribution.

> Right now, there's no way to do that - i.e. we've gone 
> backwards from a standpoint of coupling a kernel with firmware. I
> completely understand why firmware doesn't really belong in the
> kernel, so let's add this :-)

I guess a big difference between the speedtouch and the kinds of drivers
you seem to be thinking of, is that in your case there is a fairly tight
coupling between firmware and driver versions: a given driver version will
only work with a certain version of the firmware and vice-versa.  In the case
of the speedtouch, we have no control over (and not much knowledge about)
which firmware gets given to people along with their modems, so there is really
no coupling at all between firmware versions and driver versions.

> > I'm thinking of something like this: (1) redhat (or whoever) ships
> > firmware files for every driver under the sun in /lib/firmware; (2) redhat
> > wants to allow users to have a customized initrd with only essential drivers;
> > (3) the tool goes through the list of essential drivers, looks up the firmware
> > string via MODULE_FIRMWARE, finds the file in /lib/firmware, and includes it
> > in the initrd.
> 
> That kind of thing. It's not just Red Hat who benefit - anyone who
> wants to package up a kernel and do something with it will want to
> know about firmware they might need.

For this they could just read the documentation.  They'll need to anyway
just to find out where they are supposed to get the firmware from.

> Including everything in 
> /lib/firmware "just in case" is as ugly as having userspace tools with
> duplicated logic that need to understand about the internals of a
> driver module.

Don't distributions need to ship vast quantities of firmware in /lib/firmware
anyway, I mean firmware for every driver in the kernel, since they don't know
what hardware the end user may have?  So I guess you are talking about individual
users who compile their own kernels here.


Ciao,

D.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Duncan Sands <duncan.sands@math.u-psud.fr> wrote:

> > > I haven't really understood what problem this solves.  Is this just a
> > > standardised form of documentation, or are you imagining that an automatic
> > > tool will use this to auto include a minimal set of firmware files in an
> > > initrd?
> >
> > I'm imagining that the resultant modinfo output can be used by a tool
> > for anyone to package up the correct firmware to go with a given
> > driver.

> If a tool is to do the packaging, then this means that the firmware must
> already be present on the machine, for example in /lib/firmware.

Yes. Although, potentially more clever things could happen in
userspace in the future.

> that means the role of any tool is to select a subset of the files
> in /lib/firmware, for example a minimal set for inclusion in an initrd.

For example.

> Is there really any reason not to simply throw everything in
> /lib/firmware into any initrd that is created?

You /could/ just build every driver into the initrd too, and every
firmware, and... but it might be nice if it was possible to know what
should be where. Right now, we've lost that because decoupling
firmware from the kernel means that tools outside of the kernel can't
tell what firmware files should be around.

> > Right now, there's no way to do that - i.e. we've gone
> > backwards from a standpoint of coupling a kernel with firmware. I
> > completely understand why firmware doesn't really belong in the
> > kernel, so let's add this :-)
>
> I guess a big difference between the speedtouch and the kinds of drivers
> you seem to be thinking of, is that in your case there is a fairly tight
> coupling between firmware and driver versions: a given driver version will
> only work with a certain version of the firmware and vice-versa.  In the case
> of the speedtouch, we have no control over (and not much knowledge about)
> which firmware gets given to people along with their modems, so there is really
> no coupling at all between firmware versions and driver versions.

I've also repeatedly said that I don't think this really helps with
things like speedtouch since you can't distribute that firmware
anyway. This patch does help people who play nicely with the kernel
who are migrating away from shoving blobs into the kernel (quite
rightly) and who supply redistributable firmware. One example might be
the QLogic driver in my RFC.

> > That kind of thing. It's not just Red Hat who benefit - anyone who
> > wants to package up a kernel and do something with it will want to
> > know about firmware they might need.
>
> For this they could just read the documentation.  They'll need to anyway
> just to find out where they are supposed to get the firmware from.

You're assuming firmware is always on some random vendor website under
a questionable license and can't be redistributed. That wasn't my
first consideration - this doesn't really help much there, except you
get to know what you don't have :-)

Jon.

Re: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

On 4/19/06, Jon Masters <jonathan@jonmasters.org> wrote:

> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.

I (and several others who met at OLS) would really like for this to
get upstream. Can you sling that one line patch (will forward again)
into -mm please?

Jon.

Fwd: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Jon Masters]

---------- Forwarded message ----------
From: Jon Masters <jonathan@jonmasters.org>
Date: Apr 19, 2006 12:41 AM
Subject: [PATCH] MODULE_FIRMWARE for binary firmware(s)
To: akpm@osdl.org, linux-kernel@vger.kernel.org


From: Jon Masters <jcm@redhat.com>

Right now, various kernel modules are being migrated over to use
request_firmware in order to pull in binary firmware blobs from userland
when the module is loaded. This makes sense.

However, there is right now little mechanism in place to automatically
determine which binary firmware blobs must be included with a kernel in
order to satisfy the prerequisites of these drivers. This affects
vendors, but also regular users to a certain extent too.

The attached patch introduces MODULE_FIRMWARE as a mechanism for
advertising that a particular firmware file is to be loaded - it will
then show up via modinfo and could be used e.g. when packaging a kernel.

Signed-off-by: Jon Masters <jcm@redhat.com>

diff -urN linux-2.6.16.2_orig/include/linux/module.h
linux-2.6.16.2_dev/include/linux/module.h
--- linux-2.6.16.2_orig/include/linux/module.h  2006-04-07
17:56:47.000000000 +0100
+++ linux-2.6.16.2_dev/include/linux/module.h   2006-04-12
13:51:56.000000000 +0100
@@ -155,6 +155,8 @@
 */
 #define MODULE_VERSION(_version) MODULE_INFO(version, _version)

+#define MODULE_FIRMWARE(_firmware) MODULE_INFO(firmware, _firmware)
+
 /* Given an address, look for it in the exception tables */
 const struct exception_table_entry *search_exception_tables(unsigned long add);
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: Fwd: [PATCH] MODULE_FIRMWARE for binary firmware(s)

[Sam Ravnborg]

On Sat, Jul 29, 2006 at 08:06:14PM +0100, Jon Masters wrote:
> ---------- Forwarded message ----------
> From: Jon Masters <jonathan@jonmasters.org>
> Date: Apr 19, 2006 12:41 AM
> Subject: [PATCH] MODULE_FIRMWARE for binary firmware(s)
> To: akpm@osdl.org, linux-kernel@vger.kernel.org
> 
> 
> From: Jon Masters <jcm@redhat.com>
> 
> Right now, various kernel modules are being migrated over to use
> request_firmware in order to pull in binary firmware blobs from userland
> when the module is loaded. This makes sense.
> 
> However, there is right now little mechanism in place to automatically
> determine which binary firmware blobs must be included with a kernel in
> order to satisfy the prerequisites of these drivers. This affects
> vendors, but also regular users to a certain extent too.
> 
> The attached patch introduces MODULE_FIRMWARE as a mechanism for
> advertising that a particular firmware file is to be loaded - it will
> then show up via modinfo and could be used e.g. when packaging a kernel.
NAK.
Please provide some inline documentation so users knows how it is
supposed to be used etc. See MODULE_VERSION as an example.
Changelog comments is not enough documentation.

	Sam