Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Robert Dewar]

> > The only way to get from long-mode back to legacy-mode is to reset the
> > processor.  It can be done in software but you will likely lose interrupts.
> 
> Smartdrv.sys and triple-faults come back, all is forgiven!  ;)

I have an idea, perhaps we can make the keyboard controller recognize a special
command that will reset the processor :-) :-)

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Peter Finderup Lund]

On Fri, 14 Feb 2003, Robert Dewar wrote:

> > > The only way to get from long-mode back to legacy-mode is to reset the
> > > processor.  It can be done in software but you will likely lose interrupts.
> >
> > Smartdrv.sys and triple-faults come back, all is forgiven!  ;)
>
> I have an idea, perhaps we can make the keyboard controller recognize a special
> command that will reset the processor :-) :-)

Do you think there is room enough left in the ROM for the microcontroller
used as keyboard controller?  We also need to squeeze in the already
planned backwards compatibility feature to mask the A20 line thru a
special command, you know.

But maybe we could move both these features into the motherboard chipset?

-Peter

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Peter Finderup Lund]

On Fri, 14 Feb 2003, Robert Dewar wrote:

> > > The only way to get from long-mode back to legacy-mode is to reset the
> > > processor.  It can be done in software but you will likely lose interrupts.
> >
> > Smartdrv.sys and triple-faults come back, all is forgiven!  ;)
>
> I have an idea, perhaps we can make the keyboard controller recognize a special
> command that will reset the processor :-) :-)

What if there was an undocumented instruction otherwise only used for
testing and CPU in-circuit-emulation to load all the internal state of the
CPU from memory?  Maybe that would work?

-Peter

Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Andi Kleen]

[Hmm, this is becomming a FAQ]

> Switching in and out of long mode is evil enough that I don't think it
> is worth it.  And encouraging people to write good JIT compiling

Forget it. It is completely undefined in the architecture what happens
then. You'll lose interrupts and everything. Nothing for an operating
system intended to be stable.

I have no plans at all to even think about it for Linux/x86-64.

> emulators sounds much better, especially in the long run.  But it can
> be written.

For DOS even a slow emulator should be good enough. After all most
DOS Programs are written for slow machines. Bochs running on a K8
will be hopefully fast enough. If not an JIT can be written, perhaps
you can extend valgrind for it.

Or if you really rely on a DOS program executing fast you can
always boot a 32bit kernel which of course still supports vm86
in legacy mode.

-Andi

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Peter Tattam]

On Thu, 13 Feb 2003, Andi Kleen wrote:

> [Hmm, this is becomming a FAQ]
> 
> > Switching in and out of long mode is evil enough that I don't think it
> > is worth it.  And encouraging people to write good JIT compiling
> 
> Forget it. It is completely undefined in the architecture what happens
> then. You'll lose interrupts and everything. Nothing for an operating
> system intended to be stable.
> 
> I have no plans at all to even think about it for Linux/x86-64.

I have given this some thought even though the accepted wisdom is to avoid it.

As far as I can tell, there are only two critical structures which need to be
warped.  You would need to have a legacy set of page tables ready to go, and a
new IDT which is used only for legacy mode.  If an IRQ or execption happens in
legacy mode, you warp the CPU back to long mode to handle it and then warp
back.  

The concept of warping the CPU is being formalized in the plex86 project anyway
and is likely to become more common as time goes on.  (warping is replacing
GDT/IDT/LDT/CR3 etc by stubs which then warp back to the host when anything
"interesting" happens) 

So as far as I can tell, a switch to v86 mode requires reloading page tables
(this would happen on a typical task context switch anyway), and switching the
IDT. GDT,LDT and TR can stay as is since these should trap to #GP which is
handled by the IDT change. I can't see why you would want to touch the PIC or
APIC at all, and this is usually what causes the loss of interrupts when
handling this kind of thing.

The only other unknown quantity is the time it takes for the CPU to
enable/disable long mode, but with modern CPU speeds, the interrupt latency may
only be mildy affect by such a process, unless the CPU is broken in some way. 
I see no discussion in the AMD manuals regarding the cost of the mode switch,
only what AMD engineers have hinted at.

> 
> > emulators sounds much better, especially in the long run.  But it can
> > be written.
> 
> For DOS even a slow emulator should be good enough. After all most
> DOS Programs are written for slow machines. Bochs running on a K8
> will be hopefully fast enough. If not an JIT can be written, perhaps
> you can extend valgrind for it.
> 
> Or if you really rely on a DOS program executing fast you can
> always boot a 32bit kernel which of course still supports vm86
> in legacy mode.

While an emulator sounds like a good idea, it is baggage that needs to be
included.  JIT is probably overkill if the hardware can already do it.

If the use for running v86 code is infrequent, the cost in CPU cycles to change
modes may be neglible anyway.  

If it's for regular use (e.g. an MSDOS box), I am sure the scheduler could take
into account that a v86 context switch is more expensive than a normal one and
steps could be taken to avoid it.

I contend that if the thunking code is reasonably well defined and thought out,
jumping in & out of long mode might not be as big a hassle as originally
thought.

I have a need to run v86 code from ring 0, so I'm not keen to slip other
people's code in there.  This would mean I'd need to write a v86 emulator from
scratch which I think is more time than writing the warping code that I've
suggested.

I am going have a go at doing it anyway and I'll let you know my results when I
get some real hardware. 

> 
> -Andi
> 


Peter

--
Peter R. Tattam                            peter@trumpet.com
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Andi Kleen]

> I have a need to run v86 code from ring 0, so I'm not keen to slip other
...

[for the unsuspecting readers - Peter is talking about non Linux here]


> people's code in there.  This would mean I'd need to write a v86 emulator from
> scratch which I think is more time than writing the warping code that I've
> suggested.

Have you taken a look at valgrind? (http://developer.kde.org/~sewardj/)

It is a free software x86 JIT. I don't think it supports 16bit code currently,
but it probably wouldn't be too difficult to add. It wasn't primarily designed
for speed - its main application is to instrument programs - but its slowdown
compared to running on the real CPU is moderate and its certainly fast enough
for anything designed to run on DoS.

-Andi

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Eric Northup]

On Thursday 13 February 2003 07:14 pm, Peter Tattam wrote:
> On Thu, 13 Feb 2003, Andi Kleen wrote:
> > [Hmm, this is becomming a FAQ]
> >
> > > Switching in and out of long mode is evil enough that I don't think it
> > > is worth it.  And encouraging people to write good JIT compiling
> >
> > Forget it. It is completely undefined in the architecture what happens
> > then. You'll lose interrupts and everything. Nothing for an operating
> > system intended to be stable.
> >
> > I have no plans at all to even think about it for Linux/x86-64.
[snip]
>
> The only other unknown quantity is the time it takes for the CPU to
> enable/disable long mode, but with modern CPU speeds, the interrupt latency
> may only be mildy affect by such a process, unless the CPU is broken in
> some way. I see no discussion in the AMD manuals regarding the cost of the
> mode switch, only what AMD engineers have hinted at.

I think the real issue is that AMD neither recommends nor supports this 
strategy.  ( http://www.x86-64.org/lists/discuss/msg02964.html ... there were 
better posts but I couldn't find them)  People with real hardware can't talk 
about it right now, but it seems to me this is just begging to get hit by 
errata -- how much effore do you think team Hammer spent testing a subtle 
mode transition which is marked "Don't do that!" ?

> > > emulators sounds much better, especially in the long run.  But it can
> > > be written.
> >
> > For DOS even a slow emulator should be good enough. After all most
> > DOS Programs are written for slow machines. Bochs running on a K8
> > will be hopefully fast enough. If not an JIT can be written, perhaps
> > you can extend valgrind for it.
> >
> > Or if you really rely on a DOS program executing fast you can
> > always boot a 32bit kernel which of course still supports vm86
> > in legacy mode.
>
> While an emulator sounds like a good idea, it is baggage that needs to be
> included.  JIT is probably overkill if the hardware can already do it.

I am actually working on a dynamic translator for x86, and am starting with 
16-bit real-mode.  It's a bit OT for linux-kernel, and it's not done yet so 
I'll spare you the details, but the point is that the kernel doesn't need to 
do anything special to help an emulator/dynamic translator, and that it 
*shouldn't* let you run real-mode code on the hardware.

> I contend that if the thunking code is reasonably well defined and thought
> out, jumping in & out of long mode might not be as big a hassle as
> originally thought.

Even the best code is subject to the limitations of the hardware it is run on.

-Eric

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Peter Tattam]

On Thu, 13 Feb 2003, Eric Northup wrote:

> On Thursday 13 February 2003 07:14 pm, Peter Tattam wrote:
> > On Thu, 13 Feb 2003, Andi Kleen wrote:
> > > [Hmm, this is becomming a FAQ]
> > >
> > > > Switching in and out of long mode is evil enough that I don't think it
> > > > is worth it.  And encouraging people to write good JIT compiling
> > >
> > > Forget it. It is completely undefined in the architecture what happens
> > > then. You'll lose interrupts and everything. Nothing for an operating
> > > system intended to be stable.
> > >
> > > I have no plans at all to even think about it for Linux/x86-64.
> [snip]
> >
> > The only other unknown quantity is the time it takes for the CPU to
> > enable/disable long mode, but with modern CPU speeds, the interrupt latency
> > may only be mildy affect by such a process, unless the CPU is broken in
> > some way. I see no discussion in the AMD manuals regarding the cost of the
> > mode switch, only what AMD engineers have hinted at.
> 
> I think the real issue is that AMD neither recommends nor supports this 
> strategy.  ( http://www.x86-64.org/lists/discuss/msg02964.html ... there were 
> better posts but I couldn't find them)  People with real hardware can't talk 
> about it right now, but it seems to me this is just begging to get hit by 
> errata -- how much effore do you think team Hammer spent testing a subtle 
> mode transition which is marked "Don't do that!" ?
> 

well, I guess AMD need to come out & explicitly state this somewhere other than
on a mailing list.   I wouldn't be only one tempted to see if it can be done,
and if it becomes "necessary" for some OSes, AMD will get locked into a
backward compatibility minefield.  Anyone know what Windows 64 does about this
issue?  If Microsoft considers that it is sufficient to warp the CPU for v86
emulation, it may just be a done deal.

Peter

--
Peter R. Tattam                            peter@trumpet.com
Managing Director,    Trumpet Software International Pty Ltd
Hobart, Australia,  Ph. +61-3-6245-0220,  Fax +61-3-62450210

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Thomas J. Merritt]

|<><><><><> Original message from Peter Tattam  <><><><><>
|On Thu, 13 Feb 2003, Eric Northup wrote:
|
|> On Thursday 13 February 2003 07:14 pm, Peter Tattam wrote:
|> > On Thu, 13 Feb 2003, Andi Kleen wrote:
|> > > [Hmm, this is becomming a FAQ]
|> > >
|> > > > Switching in and out of long mode is evil enough that I don't think it
|> > > > is worth it.  And encouraging people to write good JIT compiling
|> > >
|> > > Forget it. It is completely undefined in the architecture what happens
|> > > then. You'll lose interrupts and everything. Nothing for an operating
|> > > system intended to be stable.
|> > >
|> > > I have no plans at all to even think about it for Linux/x86-64.
|> [snip]
|> >
|> > The only other unknown quantity is the time it takes for the CPU to
|> > enable/disable long mode, but with modern CPU speeds, the interrupt latency
|> > may only be mildy affect by such a process, unless the CPU is broken in
|> > some way. I see no discussion in the AMD manuals regarding the cost of the
|> > mode switch, only what AMD engineers have hinted at.
|> 
|> I think the real issue is that AMD neither recommends nor supports this 
|> strategy.  ( http://www.x86-64.org/lists/discuss/msg02964.html ... there were
| 
|> better posts but I couldn't find them)  People with real hardware can't talk 
|> about it right now, but it seems to me this is just begging to get hit by 
|> errata -- how much effore do you think team Hammer spent testing a subtle 
|> mode transition which is marked "Don't do that!" ?
|> 
|
|well, I guess AMD need to come out & explicitly state this somewhere other than
|on a mailing list.   I wouldn't be only one tempted to see if it can be done,
|and if it becomes "necessary" for some OSes, AMD will get locked into a
|backward compatibility minefield.  Anyone know what Windows 64 does about this
|issue?  If Microsoft considers that it is sufficient to warp the CPU for v86
|emulation, it may just be a done deal.

The only way to get from long-mode back to legacy-mode is to reset the
processor.  It can be done in software but you will likely lose interrupts.
Attempting to switch out of long-mode by modifying EFER will just get you a #GP
fault.  You might want to read Volume 2 section 14.6.2.

TJ Merritt
tjm@codegen.com
1-925-462-4300 x115

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Peter Finderup Lund]

On Thu, 13 Feb 2003, Thomas J. Merritt wrote:

> The only way to get from long-mode back to legacy-mode is to reset the
> processor.  It can be done in software but you will likely lose interrupts.

Smartdrv.sys and triple-faults come back, all is forgiven!  ;)

-Peter

Re: [discuss] Re: [Bug 350] New: i386 context switch very slow compared to 2.4 due to wrmsr (performance)

[Eric W. Biederman]

Peter Tattam <peter@jazz-1.trumpet.com.au> writes:

> On Thu, 13 Feb 2003, Andi Kleen wrote:
>
> > For DOS even a slow emulator should be good enough. After all most
> > DOS Programs are written for slow machines. Bochs running on a K8
> > will be hopefully fast enough. If not an JIT can be written, perhaps
> > you can extend valgrind for it.

valgrind certainly sounds interesting.

> > Or if you really rely on a DOS program executing fast you can
> > always boot a 32bit kernel which of course still supports vm86
> > in legacy mode.
> 
> While an emulator sounds like a good idea, it is baggage that needs to be
> included.  JIT is probably overkill if the hardware can already do it.

A good JIT supporting infrastructure is a requirement of a high quality
implementation otherwise someone will complain.  But at the same time
only the inner loops really need to be optimized.  The rest of the code
will be much less noticable.

> If the use for running v86 code is infrequent, the cost in CPU cycles to change
> modes may be neglible anyway.  
> 
> If it's for regular use (e.g. an MSDOS box), I am sure the scheduler could take
> into account that a v86 context switch is more expensive than a normal one and
> steps could be taken to avoid it.

Nope that doesn't help.  The trap rate is ruled by the number of instructions
that must be emulated.  Not by the timer.  I haven't profiled this closely but
it matches with my experience with dosemu.  Given the increasing cost of traps,
and the relative fixed frequency of instructions that require traps to
be emulated, I believe it can be shown that using the native cpu is an
increasing bad idea.

The worst case is using an ega 4 plane mode, and emulating it.  Which
is actually faster to emulate all of the instructions in than to take
traps when there are instructions you must emulate.

> I contend that if the thunking code is reasonably well defined and thought out,
> jumping in & out of long mode might not be as big a hassle as originally
> thought.

I contend that v86 mode has stunted the growth of more powerful techniques on
Linux because v86 mode is so trivial to use.  Not implementing v86 mode in
the kernel of x86-64 should encourage all of the nice techniques that need to
be built.

For most JIT recompilation all that needs to be written is a fast loop
scanning for instructions that must be emulated.  If those instructions
are not present on a page the page is good to go.  For questionable pages
you can do something else.

> I have a need to run v86 code from ring 0, so I'm not keen to slip other
> people's code in there.  This would mean I'd need to write a v86 emulator from
> scratch which I think is more time than writing the warping code that I've
> suggested.
> 
> I am going have a go at doing it anyway and I'll let you know my results when I
> get some real hardware. 

You don't want to try it in the simulator?  Is it to slow for you?

I have gone as far as testing switching in and out of long mode, and it is not
to difficult.   But I have not setup exception handlers to reflect
things from 32bit mode to 64bit mode etc.

Eric