mirror of https://lore.kernel.org/lkml/
 help / color / mirror / Atom feed
From: Chuck Lever <chuck.lever@oracle.com>
To: Sagi Grimberg <sagi@grimberg.me>
Cc: Bart Van Assche <Bart.VanAssche@wdc.com>,
	"jgg@ziepe.ca" <jgg@ziepe.ca>, "arnd@arndb.de" <arnd@arndb.de>,
	"dledford@redhat.com" <dledford@redhat.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"leonro@mellanox.com" <leonro@mellanox.com>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>
Subject: Re: [PATCH] RDMA/core: reduce IB_POLL_BATCH constant
Date: Wed, 21 Feb 2018 10:10:27 -0500	[thread overview]
Message-ID: <361D2F0F-F03C-438D-9C90-D6ABD0CAC12B@oracle.com> (raw)
In-Reply-To: <c3eab137-1c37-0e6c-a53a-df2cf80653d2@grimberg.me>



> On Feb 21, 2018, at 8:44 AM, Sagi Grimberg <sagi@grimberg.me> wrote:
> 
> 
>>> On Tue, 2018-02-20 at 21:59 +0100, Arnd Bergmann wrote:
>>>> /* # of WCs to poll for with a single call to ib_poll_cq */
>>>> -#define IB_POLL_BATCH			16
>>>> +#define IB_POLL_BATCH			8
>>> 
>>> The purpose of batch polling is to minimize contention on the cq spinlock.
>>> Reducing the IB_POLL_BATCH constant may affect performance negatively. Has
>>> the performance impact of this change been verified for all affected drivers
>>> (ib_srp, ib_srpt, ib_iser, ib_isert, NVMeOF, NVMeOF target, SMB Direct, NFS
>>> over RDMA, ...)?
>> Only the users of the DIRECT polling method use an on-stack
>> array of ib_wc's. This is only the SRP drivers.
>> The other two modes have use of a dynamically allocated array
>> of ib_wc's that hangs off the ib_cq. These shouldn't need any
>> reduction in the size of this array, and they are the common
>> case.
>> IMO a better solution would be to change ib_process_cq_direct
>> to use a smaller on-stack array, and leave IB_POLL_BATCH alone.
> 
> The only reason why I added this array on-stack was to allow consumers
> that did not use ib_alloc_cq api to call it, but that seems like a
> wrong decision when thinking it over again (as probably these users
> did not set the wr_cqe correctly).
> 
> How about we make ib_process_cq_direct use the cq wc array and add
> a WARN_ON statement (and fail it gracefully) if the caller used this
> API without calling ib_alloc_cq?

Agreed, I prefer that all three modes use dynamically allocated
memory for that array.


> --
> diff --git a/drivers/infiniband/core/cq.c b/drivers/infiniband/core/cq.c
> index bc79ca8215d7..cd3e9e124834 100644
> --- a/drivers/infiniband/core/cq.c
> +++ b/drivers/infiniband/core/cq.c
> @@ -25,10 +25,10 @@
> #define IB_POLL_FLAGS \
>        (IB_CQ_NEXT_COMP | IB_CQ_REPORT_MISSED_EVENTS)
> 
> -static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc)
> +static int __ib_process_cq(struct ib_cq *cq, int budget)
> {
>        int i, n, completed = 0;
> -       struct ib_wc *wcs = poll_wc ? : cq->wc;
> +       struct ib_wc *wcs = cq->wc;
> 
>        /*
>         * budget might be (-1) if the caller does not
> @@ -72,9 +72,9 @@ static int __ib_process_cq(struct ib_cq *cq, int budget, struct ib_wc *poll_wc)
>  */
> int ib_process_cq_direct(struct ib_cq *cq, int budget)
> {
> -       struct ib_wc wcs[IB_POLL_BATCH];
> -
> -       return __ib_process_cq(cq, budget, wcs);
> +       if (unlikely(WARN_ON_ONCE(!cq->wc)))
> +               return 0;
> +       return __ib_process_cq(cq, budget);
> }
> EXPORT_SYMBOL(ib_process_cq_direct);
> 
> @@ -88,7 +88,7 @@ static int ib_poll_handler(struct irq_poll *iop, int budget)
>        struct ib_cq *cq = container_of(iop, struct ib_cq, iop);
>        int completed;
> 
> -       completed = __ib_process_cq(cq, budget, NULL);
> +       completed = __ib_process_cq(cq, budget);
>        if (completed < budget) {
>                irq_poll_complete(&cq->iop);
>                if (ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
> @@ -108,7 +108,7 @@ static void ib_cq_poll_work(struct work_struct *work)
>        struct ib_cq *cq = container_of(work, struct ib_cq, work);
>        int completed;
> 
> -       completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE, NULL);
> +       completed = __ib_process_cq(cq, IB_POLL_BUDGET_WORKQUEUE);
>        if (completed >= IB_POLL_BUDGET_WORKQUEUE ||
>            ib_req_notify_cq(cq, IB_POLL_FLAGS) > 0)
>                queue_work(ib_comp_wq, &cq->work);
> --

--
Chuck Lever

      parent reply	other threads:[~2018-02-21 15:10 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-02-20 20:59 Arnd Bergmann
2018-02-20 21:14 ` Parav Pandit
2018-02-20 21:54   ` Arnd Bergmann
2018-02-20 21:14 ` Bart Van Assche
2018-02-20 21:47   ` Chuck Lever
2018-02-21  9:47     ` Max Gurtovoy
2018-02-21 13:44     ` Sagi Grimberg
2018-02-21 14:45       ` Max Gurtovoy
2018-02-22 15:39         ` Sagi Grimberg
2018-02-27 22:09           ` Jason Gunthorpe
2018-02-27 22:15             ` Max Gurtovoy
2018-02-28  0:21               ` Bart Van Assche
2018-02-28  9:50                 ` Max Gurtovoy
2018-02-28 18:55                   ` Doug Ledford
2018-03-01  9:36                     ` Max Gurtovoy
2018-02-21 15:10       ` Chuck Lever [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=361D2F0F-F03C-438D-9C90-D6ABD0CAC12B@oracle.com \
    --to=chuck.lever@oracle.com \
    --cc=Bart.VanAssche@wdc.com \
    --cc=arnd@arndb.de \
    --cc=dledford@redhat.com \
    --cc=jgg@ziepe.ca \
    --cc=leonro@mellanox.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=sagi@grimberg.me \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox

all inboxes | Powered by JetHome®