Re: [PATCH 2/5] rust: device: Add a minimal RawDevice trait

[Robin Murphy <robin.murphy@arm.com>]

On 2023-02-24 15:32, Greg Kroah-Hartman wrote:
> On Fri, Feb 24, 2023 at 02:32:47PM +0000, Robin Murphy wrote:
>> On 2023-02-24 14:11, Greg Kroah-Hartman wrote:
>>> Thanks for the detailed rust explainations, I'd like to just highlight
>>> one thing:
>>>
>>> On Fri, Feb 24, 2023 at 10:15:12PM +0900, Asahi Lina wrote:
>>>> On 24/02/2023 20.23, Greg Kroah-Hartman wrote:
>>>>> And again, why are bindings needed for a "raw" struct device at all?
>>>>> Shouldn't the bus-specific wrappings work better?
>>>>
>>>> Because lots of kernel subsystems need to be able to accept "any" device
>>>> and don't care about the bus! That's what this is for.
>>>
>>> That's great, but:
>>>
>>>> All the bus
>>>> wrappers would implement this so they can be used as an argument for all
>>>> those subsystems (plus a generic one when you just need to pass around
>>>> an actual owned generic reference and no longer need bus-specific
>>>> operations - you can materialize that out of a RawDevice impl, which is
>>>> when get_device() would be called). That's why I'm introducing this now,
>>>> because both io_pgtable and rtkit need to take `struct device` pointers
>>>> on the C side so we need some "generic struct device" view on the Rust side.
>>>
>>> In looking at both ftkit and io_pgtable, those seem to be good examples
>>> of how "not to use a struct device", so trying to make safe bindings
>>> from Rust to these frameworks is very ironic :)
>>>
>>> rtkit takes a struct device pointer and then never increments it,
>>> despite saving it off, which is unsafe.  It then only uses it to print
>>> out messages if things go wrong (or right in some cases), which is odd.
>>> So it can get away from using a device pointer entirely, except for the
>>> devm_apple_rtkit_init() call, which I doubt you want to call from rust
>>> code, right?
>>>
>>> for io_pgtable, that's a bit messier, you want to pass in a device that
>>> io_pgtable treats as a "device" but again, it is NEVER properly
>>> reference counted, AND, it is only needed to try to figure out the bus
>>> operations that dma memory should be allocated from for this device.  So
>>> what would be better to save off there would be a pointer to the bus,
>>> which is constant and soon will be read-only so there are no lifetime
>>> rules needed at all (see the major struct bus_type changes going into
>>> 6.3-rc1 that will enable that to happen).
>>
>> FWIW the DMA API *has* to know which specific device it's operating with,
>> since the relevant properties can and do vary even between different devices
>> within a single bus_type (e.g. DMA masks).
> 
> What bus_type has different DMA masks depending on the device on that
> bus today?

Certainly PCI and platform, which are about 99% of what matters in terms 
of DMA. You may say "ewww" again, but even on something as "normal" as a 
PCIe GPU chip, the audio codec function may well have a different DMA 
mask from the GPU function, let alone distinct devices across 
cards/slots/segments/etc. differing - e.g. some PCIe USB controllers are 
still limited to 32 bits, where your more-capable GPU/NVMe/NIC etc. 
would definitely not be cool with a lowest-common-denominator constraint.

> And the iommu ops are on the bus, not the device, but there is a
> iommu_group on the device, is that what you are referring to?
> 
> Am I getting iommu and dma stuff confused here?  A bus also has dma
> callbacks, but yet the device itself has dma_ops?

Confusion is excusable here - the bus "dma_configure" callbacks are 
mostly actually IOMMU configuration, which is a bit of historical legacy 
that still needs more cleaning up (because it makes things happen in the 
wrong order from what the IOMMU API really wants). I'm hoping to get to 
that soon once we land my current series finishing the bus->iommu_ops 
removal.

(FWIW what I want to do is flip things around so the buses just provide 
a method for the IOMMU API to to retrieve whatever bus-defined IDs it 
needs to associate with a given device, rather than having the bus code 
call into the IOMMU API itself as currently.)

>> In the case of io-pgtable at least, there's no explicit refcounting since
>> the struct device must be the one representing the physical
>> platform/PCI/etc. device consuming the pagetable, so if that were to
>> disappear from underneath its driver while the pagetable is still in use,
>> things would already have gone very very wrong indeed :)
> 
> But that could happen at any point in time, the device can be removed
> from the system with no warning, how do you guarantee that io-pgtable is
> being initialized with a device that can NOT be removed?
> 
> Think of drawers containing CPUs and PCI devices and memory, Linux
> has supported hot-removal of those for decades.  (well, not for memory,
> we could only hot-add that...)
> 
> Again, passing in a pointer to a struct device, and saving it off
> WITHOUT incrementing the reference count is not ok, that's not how
> reference counts work...

io-pgtable is just a utility library which that device's driver is 
calling into. It never does anything with that pointer by itself, and 
it's implicitly expected that the ops are only going to be called by 
whoever allocated them. If the driver needs to cope with a surprise 
removal event while bound, that's on the driver, but I'd assume the bus 
code is still expected to at least give it a chance to clean up 
gracefully before the struct device completely disappears. I don't think 
I've ever seen a driver explicitly take a reference on its own device 
for "normal" operation, so either everything's wrong, or that guarantee 
must already come from deeper in the driver core.

Yes, *technically* someone could invoke io-pgtable ops from some context 
other than the driver which allocated them, but it would be wrong and 
the answer would be "don't do that". We've got a huge quantity of APIs 
all over Linux where the expected usage model is intuitive, but 
impractical to formally assert, so it's just assumed that you do the 
right thing and expect fireworks if you don't. I do fully appreciate 
that "don't do that" is not an easy thing for Rust to nail down.

There's nothing inherently special about saving a pointer in a structure 
either - if someone's failing to respect the expected conventions, then 
the lifetime of a pointer *anywhere* matters, and even something as 
innocuous as dev_name() should strictly take a reference on its argument 
in case it's called concurrently with the device being freed. In fact, 
consider if get_device() is pre-empted right on entry, and by the time 
it comes back to dereference the kobject, someone else has racily 
dropped the last reference and already freed it. Unless we can draw a 
line of reasonable expectations somewhere, we've no pretence of being 
able to guarantee *anything*.

> But again, let's see about disconnecting the iommu ops entirely from the
> device and just relying on the bus, that should work better, rigth?

Um, other way - iommu_ops have almost finished moving *off* the bus to 
be per-device. You acked the patch the other week and seemed pleased 
with it ;)

But either way that's orthogonal to this case, since the device that 
io-pgtable deals with cannot have iommu_ops of its own (it *is* the 
IOMMU, or at least a GPU MMU acting as one).

Cheers,
Robin.