Re: [PATCH] x86: skip delays during SMP initialization similar to Xen

[Ingo Molnar <mingo@kernel.org>]

* Len Brown <lenb@kernel.org> wrote:

> On Thu, May 7, 2015 at 6:23 AM, Ingo Molnar <mingo@kernel.org> wrote:
> 
> These numbers do not look right.
> 
> > Here's what the boot time looks like on a 120 CPUs system, with the
> > patch applied:
> >
> > [    0.558947] x86: Booting SMP configuration:
> > [    0.563375] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7   #8   #9  #10  #11  #12  #13  #14
> > [    0.644851] .... node  #1, CPUs:    #15  #16  #17  #18  #19  #20  #21  #22  #23  #24  #25  #26  #27  #28  #29
> > [    0.830474] .... node  #2, CPUs:    #30  #31  #32  #33  #34  #35  #36  #37  #38  #39  #40  #41  #42  #43  #44
> > [    1.016357] .... node  #3, CPUs:    #45  #46  #47  #48  #49  #50  #51  #52  #53  #54  #55  #56  #57  #58  #59
> > [    1.202342] .... node  #0, CPUs:    #60  #61  #62  #63  #64  #65  #66  #67  #68  #69  #70  #71  #72  #73  #74
> > [    1.283864] .... node  #1, CPUs:    #75  #76  #77  #78  #79  #80  #81  #82  #83  #84  #85  #86  #87  #88  #89
> > [    1.397131] .... node  #2, CPUs:    #90  #91  #92  #93  #94  #95  #96  #97  #98  #99 #100 #101 #102 #103 #104
> > [    1.510417] .... node  #3, CPUs:   #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119
> > [    1.620967] x86: Booted up 4 nodes, 120 CPUs
> > [    1.625928] smpboot: Total of 120 processors activated (672866.16 BogoMIPS)
> >
> > 1.1 seconds to boot 120 CPUs, 10.8 seconds to hit init, that's an
> > entirely reasonable runtime I think.
> >
> > It was 20+ seconds before that, 10+ seconds for the SMP bootup
> > sequence.
> 
> (1.625928-0.558947) = 1.07 seconds to online 119 additional cpus.
> /119 = .0089662268 each, lets call it 9ms.
> 
> Here is my ivb-ex running stock fedora 21's Linux-3.19 (no patch applied):
> 
> [    0.404369] x86: Booting SMP configuration:
> [    0.409492] .... node  #0, CPUs:          #1
> [    0.439900] NMI watchdog: enabled on all CPUs, permanently consumes
> one hw-PMU counter.
> [    0.450533]    #2   #3   #4   #5   #6   #7   #8   #9  #10  #11  #12  #13  #14
> [    0.652409] .... node  #1, CPUs:    #15  #16  #17  #18  #19  #20
> #21  #22  #23  #24  #25  #26  #27  #28  #29
> [    0.999645] .... node  #2, CPUs:    #30  #31  #32  #33  #34  #35
> #36  #37  #38  #39  #40  #41  #42  #43  #44
> [    1.346991] .... node  #3, CPUs:    #45  #46  #47  #48  #49  #50
> #51  #52  #53  #54  #55  #56  #57  #58  #59
> [    1.694171] .... node  #0, CPUs:    #60  #61  #62  #63  #64  #65
> #66  #67  #68  #69  #70  #71  #72  #73  #74
> [    1.928248] .... node  #1, CPUs:    #75  #76  #77  #78  #79  #80
> #81  #82  #83  #84  #85  #86  #87  #88  #89
> [    2.198370] .... node  #2, CPUs:    #90  #91  #92  #93  #94  #95
> #96  #97  #98  #99 #100 #101 #102 #103 #104
> [    2.468574] .... node  #3, CPUs:   #105 #106 #107 #108 #109 #110
> #111 #112 #113 #114 #115 #116 #117 #118 #119
> [    2.737884] x86: Booted up 4 nodes, 120 CPUs
> [    2.743758] smpboot: Total of 120 processors activated (671097.18 BogoMIPS)
> 
> (2.743758-0.404369) = 2.339389 for all 119 processors
> /119 = .01965873109243697478 - lets call it 19ms each
> 
> so this baseline case is 19ms/processor -- which matches above, where
> we delete 10ms/processor.

> But even at 19ms each, this is totals only 2.3 seconds for all 119 
> processors. So I don't understand your reference to 10+ seconds, 
> above.

Hm, so what I was referring to is:

[    0.205318] x86: Booting SMP configuration:
[    0.206005] .... node  #0, CPUs:          #1   #2   #3   #4   #5   #6   #7   #8   #9  #10  #11  #12  #13  #14  #15  #16  #17  #18  #19  #20  #21  #22  #23  #24  #25  #26  #27  #28  #29  #30  #31  #32  #33  #34  #35  #36  #37  #38  #39  #40  #41  #42  #43  #44  #45  #46  #47  #48  #49  #50  #51  #52  #53  #54  #55  #56  #57  #58  #59  #60  #61  #62  #63  #64  #65  #66  #67  #68  #69  #70  #71  #72  #73  #74  #75  #76  #77  #78  #79  #80  #81  #82  #83  #84  #85  #86  #87  #88  #89  #90  #91  #92  #93  #94  #95  #96  #97  #98  #99 #100 #101 #102 #103 #104 #105 #106 #107 #108 #109 #110 #111 #112 #113 #114 #115 #116 #117 #118 #119
[    9.021191] x86: Booted up 1 node, 120 CPUs
[    9.021803] smpboot: Total of 120 processors activated (672102.75 BogoMIPS)
[    9.027282] devtmpfs: initialized

So it's 8.9 seconds - but still much more than you'd expect. The delay 
is evenly distributed amongst CPUs.

> ps. I wish the BIOS boot time on this box were as fast as Linux:-)

LOL! :-)

Thanks,

	Ingo