Re: [PATCH] driver core: multithreaded probing - more parallelism control

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Stefan Richter]

Huang, Ying wrote:
> This is a new version of multithreaded probing patch, with more
> parallelism control added.

Thanks.  (I'd like to try it out but will probably be busy with other
stuff during the next few weeks.)

...
> A field named "probe_queue_no" is added to "struct device", which

I'd call it probe_queue_number or maybe probe_queue_id.  The term "no"
is ambiguous.

> indicates probing queue No. on which the probing of the device will be
> done. The subsystem can control the parallelism through this field.

Is the queue number kernel-global or per subsystem?

...
> + * schedule_probe - schedule a probing to be done later
> + * @probe_queue_no: probing queue No. on which the probing will be done
> + * @probe: probing infromation include probing function and parameter
                      ^^^^^^^^^^^
typo: information
-- 
Stefan Richter
-=====-=-=== -==- =-=--
http://arcgraph.de/sr/

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Stefan Richter]

> Huang, Ying wrote:
...
>> + * @probe: probing infromation include probing function and parameter
>                       ^^^^^^^^^^^
> typo: information

Also, the meaning of the rest of the sentence is unclear.
-- 
Stefan Richter
-=====-=-=== -==- =-=--
http://arcgraph.de/sr/

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Huang, Ying]

On Wed, 2007-06-20 at 17:09 +0200, Stefan Richter wrote:
> I'd call it probe_queue_number or maybe probe_queue_id.  The term "no"
> is ambiguous.

Yes, I think probe_queue_id is better.

> Is the queue number kernel-global or per subsystem?

The queue number is kernel-global. I think this is easy to be
implemented. And the serialization demand between subsystem can be
satisfied too.

> > + * @probe: probing infromation include probing function and parameter
>                       ^^^^^^^^^^^
> typo: information

Sorry, I will correct it in the next version.

Best Regards,
Huang Ying

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Stefan Richter]

Huang, Ying wrote:
>> Is the queue number kernel-global or per subsystem?
> 
> The queue number is kernel-global.

Then there is an API required to allocate and deallocate exclusive queue
IDs.  This feels strange as a mechanism for (de-)serialization, and
might introduce some bulk WRT code and data.

Really, I don't believe there is anything else required from subsystems'
point of view than
  - the possibility to keep plain serial driver matching/probing,
  - to allow unrestricted parallelism,
  - to mark devices whose child devices shall be matched/probed
    serially.

Should there be a subsystem which has more special demands on mixture of
parallelism and serialization, it can easily use private means to
serialize certain parts of driver probes, for example with the familiar
mechanism of mutexes.

Or if need be, such a subsystem can implement its own threading model.
The FireWire subsystem for example first fetches so-called configuration
ROMs from each node on a bus by means of asynchronous split
transactions.  The ROMs are scanned for device properties and
capabilities, and then drivers are matched/probed.  The new FireWire
subsystem currently uses workqueue jobs to read the ROMs.  The old
FireWire subsystem uses one kernel thread per bus.  Before the new
FireWire subsystem was announced, I planned to let the bus kthread spawn
node kthreads which (1) fetch and scan ROMs and (2) match and probe
drivers for each unit in a node.

If the old FireWire subsystem had a future, I would most certainly not
use your mechanism but implement what you described.  I am not sure
about the new FireWire subsystem; there isn't much practical experience
with it yet.
-- 
Stefan Richter
-=====-=-=== -==- =-=-=
http://arcgraph.de/sr/

RE: [PATCH] driver core: multithreaded probing - more parallelism control

[Huang, Ying]

>-----Original Message-----
>From: Stefan Richter [mailto:stefanr@s5r6.in-berlin.de]
>> The queue number is kernel-global.
>
>Then there is an API required to allocate and deallocate exclusive
queue
>IDs.  This feels strange as a mechanism for (de-)serialization, and
>might introduce some bulk WRT code and data.

I think the queue IDs of different subsystem need not to be exclusive.
The subsystem can allocate queue IDs arbitrarily. If one queue ID is
shared between several subsystems, corresponding probing will be
serialized. This will slow down the probing unnecessarily, but there
will be no race condition.

The benefit of the mechanism is that the maximum parallelism of probing
can be control.

>Really, I don't believe there is anything else required from
subsystems'
>point of view than
>  - the possibility to keep plain serial driver matching/probing,
>  - to allow unrestricted parallelism,
>  - to mark devices whose child devices shall be matched/probed
>    serially.

Maybe.

>Should there be a subsystem which has more special demands on mixture
of
>parallelism and serialization, it can easily use private means to
>serialize certain parts of driver probes, for example with the familiar
>mechanism of mutexes.
>
>Or if need be, such a subsystem can implement its own threading model.
>The FireWire subsystem for example first fetches so-called
configuration
>ROMs from each node on a bus by means of asynchronous split
>transactions.  The ROMs are scanned for device properties and
>capabilities, and then drivers are matched/probed.  The new FireWire
>subsystem currently uses workqueue jobs to read the ROMs.  The old
>FireWire subsystem uses one kernel thread per bus.  Before the new
>FireWire subsystem was announced, I planned to let the bus kthread
spawn
>node kthreads which (1) fetch and scan ROMs and (2) match and probe
>drivers for each unit in a node.

I know nothing about FireWire. If I say something nonsense, please just
ignore it.

Whether is it possible that a new unit is inserted during probing
kthread is running. If it is possible, the probing kthread maybe ignore
the new inserted unit. So a kthread for this new unit should be created
upon inserting. And there must be some synchronization mechanism
provided. So, I think something like probing queue may be better than
raw kthread for this. And the bus scanning thread may need to exist
during system running, using the probing queue the bus scanning thread
can be created and destroyed on demanded too.

Thank you very much for your comment.

Best Regards,
Huang Ying

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Stefan Richter]

Huang, Ying wrote:
> I think the queue IDs of different subsystem need not to be exclusive.
> The subsystem can allocate queue IDs arbitrarily. If one queue ID is
> shared between several subsystems, corresponding probing will be
> serialized. This will slow down the probing unnecessarily, but there
> will be no race condition.

Parallelism between subsystems may be interesting during boot ==
"coldplug", /if/ the machine has time-consuming devices to probe on
/different/ types of buses.  Of course some machines do the really
time-consuming stuff on only one type of bus.  Granted, parallelism
betwen subsystems is not very interesting anymore later after boot ==
"hotplug".

[...]
> Whether is it possible that a new unit is inserted during probing
> kthread is running.

Nodes and units on nodes may came and go at arbitrary points in time,
and I'm sure similar things can be said about the majority of other bus
architectures or network architectures.  We take this into account.

(The old FireWire stack will re-enter the main loop of the bus scanning
thread sometime after a bus reset event signaled that nodes or units may
have appeared or disappeared.  The new FireWire stack will schedule
respective scanning workqueue jobs after such an event.)
-- 
Stefan Richter
-=====-=-=== -==- =-=-=
http://arcgraph.de/sr/

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Huang, Ying]

On Thu, 2007-06-21 at 18:21 +0200, Stefan Richter wrote:
> Parallelism between subsystems may be interesting during boot ==
> "coldplug", /if/ the machine has time-consuming devices to probe on
> /different/ types of buses.  Of course some machines do the really
> time-consuming stuff on only one type of bus.  Granted, parallelism
> betwen subsystems is not very interesting anymore later after boot ==
> "hotplug".

Yes. So I think there are two possible solution.

1. Creating one set of probing queues for each subsystem (maybe just the
subsystems need it), so the probing queue IDs are local to each
subsystem.
2. There is only one set of probing queues in whole system. The probing
queue IDs are shared between subsystems. The subsystem can select a
random starting queue ID (maybe named as start_queue_id), and allocate
the queue IDs from that point on (start_queue_id + private_queue_id). So
the probability of queue ID sharing will be reduced.

> (The old FireWire stack will re-enter the main loop of the bus scanning
> thread sometime after a bus reset event signaled that nodes or units may
> have appeared or disappeared.  The new FireWire stack will schedule
> respective scanning workqueue jobs after such an event.)

I think the workqueue is better than kernel thread here. With kernel
thread, the nodes and units may be needed to be scanned again and again
for each unit/node if many units/nodes are appeared at almost the same
time, while with workqueue, just schedule the jobs needed.

And a workqueue like the probing queue whose thread can be
created/destroyed on demand will save more resources than ordinary
workqueue. :)

Best Regards,
Huang Ying

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Cornelia Huck]

On Fri, 22 Jun 2007 09:52:38 +0000,
"Huang, Ying" <ying.huang@intel.com> wrote:

> On Thu, 2007-06-21 at 18:21 +0200, Stefan Richter wrote:
> > Parallelism between subsystems may be interesting during boot ==
> > "coldplug", /if/ the machine has time-consuming devices to probe on
> > /different/ types of buses.  Of course some machines do the really
> > time-consuming stuff on only one type of bus.  Granted, parallelism
> > betwen subsystems is not very interesting anymore later after boot ==
> > "hotplug".
> 
> Yes. So I think there are two possible solution.
> 
> 1. Creating one set of probing queues for each subsystem (maybe just the
> subsystems need it), so the probing queue IDs are local to each
> subsystem.
> 2. There is only one set of probing queues in whole system. The probing
> queue IDs are shared between subsystems. The subsystem can select a
> random starting queue ID (maybe named as start_queue_id), and allocate
> the queue IDs from that point on (start_queue_id + private_queue_id). So
> the probability of queue ID sharing will be reduced.

What should also be considered here is that we may want to have
different numbers of queues per subsystem (fewer for those where
probing is resource-heavy), but may want to restrict the total number
of queues as well. Some throttling mechanism may be helpful here (so
that a single subsystem cannot hog all queues while another is stuck
with a single queue, or double usage of queues). In fact, throttling
may be interesting for any subsystem using parallelism, especially if
the number of devices may be huge and/or probing resource hungry.

Cornelia

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Greg KH]

On Wed, Jun 20, 2007 at 09:00:58PM +0000, Huang, Ying wrote:
> Hi,
> 
> This is a new version of multithreaded probing patch, with more
> parallelism control added.
> 
> There are more control over which devices and drivers will be probed
> parallelized or serially. For example, in IEEE1394 subsystem, the
> different "units" in one "node" can be probed serially while the
> different "nodes" can be probed parallelized.
> 
> The number of threads can be controlled through a kernel command line
> parameters.
> 
> The patch is against 2.6.22-rc5. The "wait_for_probes" function in the
> patch comes from the original multithreaded probing patch. If I need do
> anything because of it, please let me know.
> 
> Any comment is welcome.

I'm still not convinced that we need to add this kind of complexity to
the driver core, instead of just letting the individual driver
subsystems do this, if they want to do it.

Especially as no subsystem wants to do this today :)

thanks,

greg k-h

Re: [PATCH] driver core: multithreaded probing - more parallelism control

[Stefan Richter]

Greg KH wrote:
> I'm still not convinced that we need to add this kind of complexity to
> the driver core, instead of just letting the individual driver
> subsystems do this, if they want to do it.
> 
> Especially as no subsystem wants to do this today :)

Yes, it should first be shown (with subsystem conversions and runtime
tests with, say, at least two different bus architectures) that features
like this really are appropriately implemented as a driver core abstraction.

I would lend a hand to put this to test, but I won't do this anymore
with the old ieee1394 subsystem, and work on the new firewire subsystem
will be focused on stabilization and feature completion in the short to
mid term.
-- 
Stefan Richter
-=====-=-=== -==- ==---
http://arcgraph.de/sr/

RE: [PATCH] driver core: multithreaded probing - more parallelismcontrol

[Huang, Ying]

>From: Greg KH [mailto:greg@kroah.com]
>I'm still not convinced that we need to add this kind of complexity to
>the driver core, instead of just letting the individual driver
>subsystems do this, if they want to do it.

It may appear not necessary that providing more multithreaded device
probing in the driver core, but it seems more necessary that providing
more parallel control in the driver core to make some device probing
more single-threaded.

There does exist multithreaded device probing in current driver core
implementation, supposing two devices are hot-plugged at the same time.
But, many device drivers are written without this taken into account. I
think it may be better to make default device probing process more
single-threaded in the driver core. The single-thread workqueue or some
customized version of workqueue like that implemented by my patch can be
used for this. The parallel control mechanism can be used to implement
multithreaded device probing in needed subsystems too.

Best Regards,
Huang, Ying

Re: [PATCH] driver core: multithreaded probing - more parallelismcontrol

[Greg KH]

On Sun, Jun 24, 2007 at 11:04:13PM +0800, Huang, Ying wrote:
> >From: Greg KH [mailto:greg@kroah.com]
> >I'm still not convinced that we need to add this kind of complexity to
> >the driver core, instead of just letting the individual driver
> >subsystems do this, if they want to do it.
> 
> It may appear not necessary that providing more multithreaded device
> probing in the driver core, but it seems more necessary that providing
> more parallel control in the driver core to make some device probing
> more single-threaded.
> 
> There does exist multithreaded device probing in current driver core
> implementation, supposing two devices are hot-plugged at the same time.

No, that is a bus-specific thing, and no bus that I know of supports
that at this time.

> But, many device drivers are written without this taken into account.

That's why no bus does this :)

> I think it may be better to make default device probing process more
> single-threaded in the driver core. The single-thread workqueue or some
> customized version of workqueue like that implemented by my patch can be
> used for this. The parallel control mechanism can be used to implement
> multithreaded device probing in needed subsystems too.

But remember, the individual busses already do this all in a single
thread anyway, nothing is needed in the driver core to do this.

thanks,

greg k-h

Re: [PATCH] driver core: multithreaded probing - more parallelismcontrol

[Cornelia Huck]

On Mon, 25 Jun 2007 01:16:24 -0700,
Greg KH <greg@kroah.com> wrote:

[I'm a bit late to the party, but...]

> On Sun, Jun 24, 2007 at 11:04:13PM +0800, Huang, Ying wrote:

> > There does exist multithreaded device probing in current driver core
> > implementation, supposing two devices are hot-plugged at the same time.
> 
> No, that is a bus-specific thing, and no bus that I know of supports
> that at this time.

The s390 channel subsystem busses should be fine with any parallelism,
especially as the css bus kicks off tons of probes (device recognition)
at the same time. Any ccw driver must be able to be handle to be called
for many devices in parallel as well (like, when someone attaches their
shiny new storage subsystem to the LPAR and some thousands of dasds
become available).

> 
> > But, many device drivers are written without this taken into account.
> 
> That's why no bus does this :)

It is possible for busses for a small set of device drivers (like the
s390 busses; maybe there are others). It looks like a bad idea to try
this for PCI :)

> 
> > I think it may be better to make default device probing process more
> > single-threaded in the driver core. The single-thread workqueue or some
> > customized version of workqueue like that implemented by my patch can be
> > used for this. The parallel control mechanism can be used to implement
> > multithreaded device probing in needed subsystems too.
> 
> But remember, the individual busses already do this all in a single
> thread anyway, nothing is needed in the driver core to do this.

I think I could make good use of some more parallelism control (for
throttling or so). Not sure if it should really sit at the driver
core level, but that would avoid reinventing the wheel.

[Goes reading the original patch]

Cornelia