mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Elliot Berman <quic_eberman@quicinc.com>
To: Jassi Brar <jassisinghbrar@gmail.com>
Cc: Bjorn Andersson <quic_bjorande@quicinc.com>,
	Murali Nalajala <quic_mnalajal@quicinc.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Krzysztof Kozlowski <krzysztof.kozlowski+dt@linaro.org>,
	Rob Herring <robh+dt@kernel.org>,
	Trilok Soni <quic_tsoni@quicinc.com>,
	"Srivatsa Vaddagiri" <quic_svaddagi@quicinc.com>,
	Carl van Schaik <quic_cvanscha@quicinc.com>,
	Prakruthi Deepak Heragu <quic_pheragu@quicinc.com>,
	Andy Gross <agross@kernel.org>,
	Dmitry Baryshkov <dmitry.baryshkov@linaro.org>,
	<linux-arm-kernel@lists.infradead.org>,
	"Mark Rutland" <mark.rutland@arm.com>,
	Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>,
	Sudeep Holla <sudeep.holla@arm.com>,
	"Marc Zyngier" <maz@kernel.org>, Jonathan Corbet <corbet@lwn.net>,
	Will Deacon <will@kernel.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Arnd Bergmann <arnd@arndb.de>,
	Srinivas Kandagatla <srinivas.kandagatla@linaro.org>,
	"Amol Maheshwari" <amahesh@qti.qualcomm.com>,
	Kalle Valo <kvalo@kernel.org>, <devicetree@vger.kernel.org>,
	<linux-doc@vger.kernel.org>, <linux-arm-msm@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v6 02/21] dt-bindings: Add binding for gunyah hypervisor
Date: Thu, 3 Nov 2022 12:45:39 -0700	[thread overview]
Message-ID: <4bf14665-3303-2449-a5d7-5b7cf01ccc29@quicinc.com> (raw)
In-Reply-To: <CABb+yY2fBa3up8Byu8axagwTEoidW+mbwene1qFyh=qJxPLJ=Q@mail.gmail.com>



On 11/2/2022 8:21 PM, Jassi Brar wrote:
> On Wed, Nov 2, 2022 at 6:23 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>
>>
>>
>> On 11/2/2022 11:24 AM, Jassi Brar wrote:
>>> On Wed, Nov 2, 2022 at 1:06 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>>
>>>> Hi Jassi,
>>>>
>>>> On 11/1/2022 7:01 PM, Jassi Brar wrote:
>>>>> On Tue, Nov 1, 2022 at 7:12 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 11/1/2022 2:58 PM, Jassi Brar wrote:
>>>>>>> On Tue, Nov 1, 2022 at 3:35 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 11/1/2022 9:23 AM, Jassi Brar wrote:
>>>>>>>>> On Mon, Oct 31, 2022 at 10:20 PM Elliot Berman <quic_eberman@quicinc.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Jassi,
>>>>>>>>>>
>>>>>>>>>> On 10/27/2022 7:33 PM, Jassi Brar wrote:
>>>>>>>>>>       > On Wed, Oct 26, 2022 at 1:59 PM Elliot Berman
>>>>>>>>>> <quic_eberman@quicinc.com> wrote:
>>>>>>>>>>       > .....
>>>>>>>>>>       >> +
>>>>>>>>>>       >> +        gunyah-resource-mgr@0 {
>>>>>>>>>>       >> +            compatible = "gunyah-resource-manager-1-0",
>>>>>>>>>> "gunyah-resource-manager";
>>>>>>>>>>       >> +            interrupts = <GIC_SPI 3 IRQ_TYPE_EDGE_RISING>, /* TX
>>>>>>>>>> full IRQ */
>>>>>>>>>>       >> +                         <GIC_SPI 4 IRQ_TYPE_EDGE_RISING>; /* RX
>>>>>>>>>> empty IRQ */
>>>>>>>>>>       >> +            reg = <0x00000000 0x00000000>, <0x00000000 0x00000001>;
>>>>>>>>>>       >> +                  /* TX, RX cap ids */
>>>>>>>>>>       >> +        };
>>>>>>>>>>       >>
>>>>>>>>>>       > All these resources are used only by the mailbox controller driver.
>>>>>>>>>>       > So, this should be the mailbox controller node, rather than the
>>>>>>>>>>       > mailbox user.> One option is to load gunyah-resource-manager as a
>>>>>>>>>> module that relies
>>>>>>>>>>       > on the gunyah-mailbox provider. That would also avoid the "Allow
>>>>>>>>>>       > direct registration to a channel" hack patch.
>>>>>>>>>>
>>>>>>>>>> A message queue to another guest VM wouldn't be known at boot time and
>>>>>>>>>> thus couldn't be described on the devicetree.
>>>>>>>>>>
>>>>>>>>> I think you need to implement of_xlate() ... or please tell me what
>>>>>>>>> exactly you need to specify in the dt.
>>>>>>>>
>>>>>>>> Dynamically created virtual machines can't be known on the dt, so there
>>>>>>>> is nothing to specify in the DT. There couldn't be a devicetree node for
>>>>>>>> the message queue client because that client is only exists once the VM
>>>>>>>> is created by userspace.
>>>>>>>>
>>>>>>> The underlying "physical channel" is the synchronous SMC instruction,
>>>>>>> which remains 1 irrespective of the number of mailbox instances
>>>>>>> created.
>>>>>>
>>>>>> I disagree that the physical channel is the SMC instruction. Regardless
>>>>>> though, there are num_online_cpus() "physical channels" with this
>>>>>> perspective.
>>>>>>
>>>>>>> So basically you are sharing one resource among users. Why doesn't the
>>>>>>> RM request the "smc instruction" channel once and share it among
>>>>>>> users?
>>>>>>
>>>>>> I suppose in this scenario, a single mailbox channel would represent all
>>>>>> message queues? This would cause Linux to serialize *all* message queue
>>>>>> hypercalls. Sorry, I can only think negative implications.
>>>>>>
>>>>>> Error handling needs to move into clients: if a TX message queue becomes
>>>>>> full or an RX message queue becomes empty, then we'll need to return
>>>>>> error back to the client right away. The clients would need to register
>>>>>> for the RTS/RTR interrupts to know when to send/receive messages and
>>>>>> have retry error handling. If the mailbox controller retried for the
>>>>>> clients as currently proposed, then we could get into a scenario where a
>>>>>> message queue could never be ready to send/receive and thus stuck
>>>>>> forever trying to process that message. The effect here would be that
>>>>>> the mailbox controller becomes a wrapper to some SMC instructions that
>>>>>> aren't related at the SMC instruction level.
>>>>>>
>>>>>> A single channel would limit performance of SMP systems because only one
>>>>>> core could send/receive a message. There is no such limitation for
>>>>>> message queues to behave like this.
>>>>>>
>>>>> This is just an illusion. If Gunyah can handle multiple calls from a
>>>>> VM parallely, even with the "bind-client-to-channel" hack you can't
>>>>> make sure different channels run on different cpu cores.  If you are
>>>>> ok with that, you could simply populate a mailbox controller with N
>>>>> channels and allocate them in any order the clients ask.
>>>>
>>>> I wanted to make sure I understood the ask here completely. On what
>>>> basis is N chosen? Who would be the mailbox clients?
>>>>
>>> A channel structure is cheap, so any number that is not likely to run
>>> out. Say you have 10 possible users in a VM, set N=16. I know ideally
>>> it should be precise and flexible but the gain in simplicity makes the
>>> trade-off very acceptable.
>>
>> I think I get the direction you are thinking now. N is chosen based off
>> of how many clients there might be. One mailbox controller will
>> represent all message queues and each channel will be one message queue.
>> There are some limitations that might make it more complex to implement
>> than having 1 message queue per controller like I have now.
>>
>> My interpretation is that mailbox controller knows the configuration of
>> its channels before being bound to a client. For dynamically created
>> message queues, the client would need tell the controller about the
>> message queue configuration. I didn't find example where client is
>> providing information about a channel to the controller.
>>
>>    1. need a mechanism to allow the client to provide the
>> gunyah_resources for the channel (i.e. the irqs and cap ids).
>>
> IIUC there is exactly one resource-manager in a VM. Right?
> Looking at your code, TX and RX irq are used only by the mailbox
> driver and are the same for all clients/users. So that should be a
> property under the mailbox controller node. Not sure what cap ids are.
> 

Ah -- "message queues" are a generic inter-VM communication mechanism 
offered by Gunyah. One use case for message queues is to communicate 
with the resource-manager, but other message queues can exist between 
other virtual machines. Those other message queues use different TX and 
RX irq and have different client protocols.

In mailbox terminology, we have one known channel at boot-up time (the 
resource manager). That known channel can inform Linux about other 
channels at runtime. The client (not the controller) decodes received 
data from the channel to discover the new channels.

One approach we found was coming from pcc.c, which has their own 
request_channel function (pcc_mbox_request_channel). We could follow 
this approach as well...

>>    2. Still need to have bind-client-to-channel patch since clients
>> aren't real devices and so shouldn't be on DT.
>>
> the clients may be virtual (serial, gpio etc) but the resource-manager
> requires some mailbox hardware to communicate, so the resource-manager
> should be the mailbox client (that further spawns virtual devices)

Yes, this the design I'm aiming for. Also want to highlight that the 
resource-manager spawns Gunyah virtual devices such as message queue 
channels.

Thanks,
Elliot

  reply	other threads:[~2022-11-03 19:45 UTC|newest]

Thread overview: 68+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-10-26 18:58 [PATCH v6 00/21] Drivers " Elliot Berman
2022-10-26 18:58 ` [PATCH v6 01/21] docs: gunyah: Introduce Gunyah Hypervisor Elliot Berman
2022-11-02 12:35   ` Bagas Sanjaya
2022-10-26 18:58 ` [PATCH v6 02/21] dt-bindings: Add binding for gunyah hypervisor Elliot Berman
2022-10-27 19:57   ` Krzysztof Kozlowski
2022-10-28  2:33   ` Jassi Brar
2022-11-01  3:19     ` Elliot Berman
2022-11-01 16:23       ` Jassi Brar
2022-11-01 20:35         ` Elliot Berman
2022-11-01 21:58           ` Jassi Brar
2022-11-02  0:12             ` Elliot Berman
2022-11-02  2:01               ` Jassi Brar
2022-11-02 18:05                 ` Elliot Berman
2022-11-02 18:24                   ` Jassi Brar
2022-11-02 23:23                     ` Elliot Berman
2022-11-03  3:21                       ` Jassi Brar
2022-11-03 19:45                         ` Elliot Berman [this message]
2022-10-26 18:58 ` [PATCH v6 03/21] gunyah: Common types and error codes for Gunyah hypercalls Elliot Berman
2022-10-26 19:47   ` Dmitry Baryshkov
2022-10-26 18:58 ` [PATCH v6 04/21] arm64: smccc: Include alternative-macros.h Elliot Berman
2022-10-26 19:46   ` Dmitry Baryshkov
2022-10-26 20:23     ` Elliot Berman
2022-10-26 20:39       ` Dmitry Baryshkov
2022-10-26 18:58 ` [PATCH v6 05/21] virt: gunyah: Add hypercalls to identify Gunyah Elliot Berman
2022-10-26 18:58 ` [PATCH v6 06/21] virt: gunyah: Identify hypervisor version Elliot Berman
2022-10-26 18:58 ` [PATCH v6 07/21] mailbox: Allow direct registration to a channel Elliot Berman
2022-10-26 18:58 ` [PATCH v6 08/21] virt: gunyah: msgq: Add hypercalls to send and receive messages Elliot Berman
2022-10-26 18:58 ` [PATCH v6 09/21] mailbox: Add Gunyah message queue mailbox Elliot Berman
2022-10-27 13:55   ` Pavan Kondeti
2022-11-01 17:44     ` Elliot Berman
2022-10-26 18:58 ` [PATCH v6 10/21] gunyah: rsc_mgr: Add resource manager RPC core Elliot Berman
2022-11-01 18:02   ` Greg Kroah-Hartman
2022-11-02  0:12     ` Elliot Berman
2022-11-02  2:53       ` Greg Kroah-Hartman
2022-11-02 18:04         ` Elliot Berman
2022-11-03  0:22           ` Greg Kroah-Hartman
2022-11-03 22:07             ` Elliot Berman
2022-11-03 22:09             ` Elliot Berman
2022-10-26 18:58 ` [PATCH v6 11/21] gunyah: rsc_mgr: Add subdevices bus Elliot Berman
2022-10-26 18:58 ` [PATCH v6 12/21] gunyah: rsc_mgr: Add VM lifecycle RPC Elliot Berman
2022-10-26 18:58 ` [PATCH v6 13/21] gunyah: vm_mgr: Introduce basic VM Manager Elliot Berman
2022-11-02  5:14   ` Greg Kroah-Hartman
2022-11-02 18:45     ` Elliot Berman
2022-11-03  0:24       ` Greg Kroah-Hartman
2022-11-04  0:11         ` Elliot Berman
2022-11-04  8:10           ` Arnd Bergmann
2022-11-04 22:38             ` Elliot Berman
2022-11-05  4:19               ` Trilok Soni
2022-11-11  0:03                 ` Elliot Berman
2022-11-11  6:24                   ` Greg Kroah-Hartman
2022-11-11 17:08                     ` Elliot Berman
2022-11-02  7:31   ` Arnd Bergmann
2022-11-02 18:44     ` Elliot Berman
2022-11-03  0:20       ` Greg Kroah-Hartman
2022-11-03 22:33         ` Elliot Berman
2022-11-03  9:39       ` Arnd Bergmann
2022-11-03 22:10         ` Elliot Berman
2022-10-26 18:58 ` [PATCH v6 14/21] gunyah: rsc_mgr: Add RPC for sharing memory Elliot Berman
2022-10-26 18:58 ` [PATCH v6 15/21] gunyah: vm_mgr: Add/remove user memory regions Elliot Berman
2022-10-26 18:58 ` [PATCH v6 16/21] gunyah: vm_mgr: Add ioctls to support basic non-proxy VM boot Elliot Berman
2022-10-26 18:58 ` [PATCH v6 17/21] samples: Add sample userspace Gunyah VM Manager Elliot Berman
2022-10-26 18:58 ` [PATCH v6 18/21] gunyah: rsc_mgr: Add platform ops on mem_lend/mem_reclaim Elliot Berman
2022-10-26 18:58 ` [PATCH v6 19/21] firmware: qcom_scm: Use fixed width src vm bitmap Elliot Berman
2022-10-26 18:58 ` [PATCH v6 20/21] firmware: qcom_scm: Register Gunyah platform ops Elliot Berman
2022-10-26 21:32   ` kernel test robot
2022-10-26 18:58 ` [PATCH v6 21/21] docs: gunyah: Document Gunyah VM Manager Elliot Berman
2022-11-02 13:05   ` Bagas Sanjaya
2022-11-02 18:04     ` Elliot Berman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=4bf14665-3303-2449-a5d7-5b7cf01ccc29@quicinc.com \
    --to=quic_eberman@quicinc.com \
    --cc=agross@kernel.org \
    --cc=amahesh@qti.qualcomm.com \
    --cc=arnd@arndb.de \
    --cc=catalin.marinas@arm.com \
    --cc=corbet@lwn.net \
    --cc=devicetree@vger.kernel.org \
    --cc=dmitry.baryshkov@linaro.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=jassisinghbrar@gmail.com \
    --cc=krzysztof.kozlowski+dt@linaro.org \
    --cc=kvalo@kernel.org \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-arm-msm@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=lorenzo.pieralisi@arm.com \
    --cc=mark.rutland@arm.com \
    --cc=maz@kernel.org \
    --cc=quic_bjorande@quicinc.com \
    --cc=quic_cvanscha@quicinc.com \
    --cc=quic_mnalajal@quicinc.com \
    --cc=quic_pheragu@quicinc.com \
    --cc=quic_svaddagi@quicinc.com \
    --cc=quic_tsoni@quicinc.com \
    --cc=robh+dt@kernel.org \
    --cc=srinivas.kandagatla@linaro.org \
    --cc=sudeep.holla@arm.com \
    --cc=will@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®