Use (two) different compilers at build-time?

Use (two) different compilers at build-time?

[Sedat Dilek]

Hi,

is it possible to use a different compiler at build-time?

I have here a problem with mm/percpu.c and wanted to build everything
with LLVM/Clang but this single file with GCC.

Is that possible?

Regards,
- Sedat -

Re: Use (two) different compilers at build-time?

[Linus Torvalds]

On Mon, Sep 7, 2015 at 12:12 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> is it possible to use a different compiler at build-time?
>
> I have here a problem with mm/percpu.c and wanted to build everything
> with LLVM/Clang but this single file with GCC.
>
> Is that possible?

It should work fine. It's occasionally how people bisect compiler bugs
in the kernel (although then it's usually just two different versions
of the same compiler), and I think people have done that with llvm
too. After all, llvm object files link to gcc-produced object files in
user space (ie libraries are most often compiled with different
compilers).

So no _guarantees_ (especially since the kernel sometimes does odd
things), but there's no overwhelming reason it shouldn't work, and
there are several reasons it should be fine.

                  Linus

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Mon, Sep 7, 2015 at 10:15 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Sep 7, 2015 at 12:12 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> is it possible to use a different compiler at build-time?
>>
>> I have here a problem with mm/percpu.c and wanted to build everything
>> with LLVM/Clang but this single file with GCC.
>>
>> Is that possible?
>
> It should work fine. It's occasionally how people bisect compiler bugs
> in the kernel (although then it's usually just two different versions
> of the same compiler), and I think people have done that with llvm
> too. After all, llvm object files link to gcc-produced object files in
> user space (ie libraries are most often compiled with different
> compilers).
>
> So no _guarantees_ (especially since the kernel sometimes does odd
> things), but there's no overwhelming reason it shouldn't work, and
> there are several reasons it should be fine.
>

OK.

So, how do I do that conveniently?

In case of percpu.o - hack in the mm/Makefile?

So, I have a percpu.o compiled with clang in my build-dir and want to
recompile just that single file with gcc and create a new bzImage.

Hope I could explain it more exactly.

- Sedat -

Re: Use (two) different compilers at build-time?

[Linus Torvalds]

On Mon, Sep 7, 2015 at 1:31 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> So, how do I do that conveniently?

For a single file? Just compile everything with the primary compiler,
and then delete the single object file, and do "make" again.

Use a wrapper around the compiler (and point to that wrapper with the
"to switch compilers from under the make, without the build paths
changing (because otherwise our makefile auto-machinery notices that
flags and command changed).

Use CC (or CROSS_COMPILE) to point at your wrapper.

What we've occasionally wanted to do for compiler bugs is to have
something that can do the above not for single files, but some kind of
file bisection thing (so bisect not in history, but in the list of
files compiled). Sadly I don't think anybody has ever made that work.

             Linus

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Mon, Sep 7, 2015 at 10:42 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Sep 7, 2015 at 1:31 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> So, how do I do that conveniently?
>
> For a single file? Just compile everything with the primary compiler,
> and then delete the single object file, and do "make" again.
>

That does not work.

I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
re-invoked make - this starts a complete new build from scratch.

$ cd $KERNEL_BUILD_DIR
$ mv mm/percpu.o mm/percpu.o_CLANG
$ make CC=gcc HOSTCC=gcc

> Use a wrapper around the compiler (and point to that wrapper with the
> "to switch compilers from under the make, without the build paths
> changing (because otherwise our makefile auto-machinery notices that
> flags and command changed).
>
> Use CC (or CROSS_COMPILE) to point at your wrapper.
>

No idea how to realize that, sorry.

> What we've occasionally wanted to do for compiler bugs is to have
> something that can do the above not for single files, but some kind of
> file bisection thing (so bisect not in history, but in the list of
> files compiled). Sadly I don't think anybody has ever made that work.
>

Sounds like a nice feature.

- Sedat -

Re: Use (two) different compilers at build-time?

[Linus Torvalds]

On Mon, Sep 7, 2015 at 1:53 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> That does not work.

.. because you didn't do what I told you to do.

> I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
> re-invoked make - this starts a complete new build from scratch.

Right. Because you changed the compiler name, so now the build system
realizes that the old build instructions are stale.

Which is why you have to:

>> Use a wrapper around the compiler (and point to that wrapper with the
>> "to switch compilers from under the make, without the build paths
>> changing (because otherwise our makefile auto-machinery notices that
>> flags and command changed).
>>
>> Use CC (or CROSS_COMPILE) to point at your wrapper.
>
> No idea how to realize that, sorry.

Literally just do something like this:

 - have a shell script call "mycompiler" and make it do gcc/llvm "$@".

 - or even just use a symlink (the script has the advantage that you
can play with the options etc too)

 - change the shell script (or symlink) itself, and make sure to use
the same CC for "make" at all times, so that the build script never
sees that the underlying command is now different.

It should work fine, I've done it a couple of times (although
admittedly not recently)

                  Linus

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Mon, Sep 7, 2015 at 10:59 PM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Mon, Sep 7, 2015 at 1:53 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>
>> That does not work.
>
> .. because you didn't do what I told you to do.
>
>> I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
>> re-invoked make - this starts a complete new build from scratch.
>
> Right. Because you changed the compiler name, so now the build system
> realizes that the old build instructions are stale.
>
> Which is why you have to:
>
>>> Use a wrapper around the compiler (and point to that wrapper with the
>>> "to switch compilers from under the make, without the build paths
>>> changing (because otherwise our makefile auto-machinery notices that
>>> flags and command changed).
>>>
>>> Use CC (or CROSS_COMPILE) to point at your wrapper.
>>
>> No idea how to realize that, sorry.
>
> Literally just do something like this:
>
>  - have a shell script call "mycompiler" and make it do gcc/llvm "$@".
>
>  - or even just use a symlink (the script has the advantage that you
> can play with the options etc too)
>
>  - change the shell script (or symlink) itself, and make sure to use
> the same CC for "make" at all times, so that the build script never
> sees that the underlying command is now different.
>
> It should work fine, I've done it a couple of times (although
> admittedly not recently)
>

OK, I have created a mycompiler shell-script and use that for CC and
HOSTCC in my own build-script.

Using CLANG...

[ /usr/bin/mycompiler ]

#!/bin/bash

clang "$@"
- EOF -

$ mycompiler --version
clang version 3.7.0 (tags/RELEASE_370/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix

Switching to GCC...

[ /usr/bin/mycompiler ]

#!/bin/bash

gcc-4.9 "$@"
- EOF -

$ mycompiler --version
gcc-4.9 (Ubuntu 4.9.2-0ubuntu1~12.04) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks, that helped me a lot.

- Sedat -

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Mon, Sep 7, 2015 at 11:20 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> On Mon, Sep 7, 2015 at 10:59 PM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>> On Mon, Sep 7, 2015 at 1:53 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>
>>> That does not work.
>>
>> .. because you didn't do what I told you to do.
>>
>>> I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
>>> re-invoked make - this starts a complete new build from scratch.
>>
>> Right. Because you changed the compiler name, so now the build system
>> realizes that the old build instructions are stale.
>>
>> Which is why you have to:
>>
>>>> Use a wrapper around the compiler (and point to that wrapper with the
>>>> "to switch compilers from under the make, without the build paths
>>>> changing (because otherwise our makefile auto-machinery notices that
>>>> flags and command changed).
>>>>
>>>> Use CC (or CROSS_COMPILE) to point at your wrapper.
>>>
>>> No idea how to realize that, sorry.
>>
>> Literally just do something like this:
>>
>>  - have a shell script call "mycompiler" and make it do gcc/llvm "$@".
>>
>>  - or even just use a symlink (the script has the advantage that you
>> can play with the options etc too)
>>
>>  - change the shell script (or symlink) itself, and make sure to use
>> the same CC for "make" at all times, so that the build script never
>> sees that the underlying command is now different.
>>
>> It should work fine, I've done it a couple of times (although
>> admittedly not recently)
>>
>
> OK, I have created a mycompiler shell-script and use that for CC and
> HOSTCC in my own build-script.
>
> Using CLANG...
>
> [ /usr/bin/mycompiler ]
>
> #!/bin/bash
>
> clang "$@"
> - EOF -
>
> $ mycompiler --version
> clang version 3.7.0 (tags/RELEASE_370/final)
> Target: x86_64-unknown-linux-gnu
> Thread model: posix
>
> Switching to GCC...
>
> [ /usr/bin/mycompiler ]
>
> #!/bin/bash
>
> gcc-4.9 "$@"
> - EOF -
>
> $ mycompiler --version
> gcc-4.9 (Ubuntu 4.9.2-0ubuntu1~12.04) 4.9.2
> Copyright (C) 2014 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>
> Thanks, that helped me a lot.
>

Sadly, this trick does not work here with Linux v4.2.

- Sedat -

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Tue, Sep 8, 2015 at 12:30 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> On Mon, Sep 7, 2015 at 11:20 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> On Mon, Sep 7, 2015 at 10:59 PM, Linus Torvalds
>> <torvalds@linux-foundation.org> wrote:
>>> On Mon, Sep 7, 2015 at 1:53 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>
>>>> That does not work.
>>>
>>> .. because you didn't do what I told you to do.
>>>
>>>> I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
>>>> re-invoked make - this starts a complete new build from scratch.
>>>
>>> Right. Because you changed the compiler name, so now the build system
>>> realizes that the old build instructions are stale.
>>>
>>> Which is why you have to:
>>>
>>>>> Use a wrapper around the compiler (and point to that wrapper with the
>>>>> "to switch compilers from under the make, without the build paths
>>>>> changing (because otherwise our makefile auto-machinery notices that
>>>>> flags and command changed).
>>>>>
>>>>> Use CC (or CROSS_COMPILE) to point at your wrapper.
>>>>
>>>> No idea how to realize that, sorry.
>>>
>>> Literally just do something like this:
>>>
>>>  - have a shell script call "mycompiler" and make it do gcc/llvm "$@".
>>>
>>>  - or even just use a symlink (the script has the advantage that you
>>> can play with the options etc too)
>>>
>>>  - change the shell script (or symlink) itself, and make sure to use
>>> the same CC for "make" at all times, so that the build script never
>>> sees that the underlying command is now different.
>>>
>>> It should work fine, I've done it a couple of times (although
>>> admittedly not recently)
>>>
>>
>> OK, I have created a mycompiler shell-script and use that for CC and
>> HOSTCC in my own build-script.
>>
>> Using CLANG...
>>
>> [ /usr/bin/mycompiler ]
>>
>> #!/bin/bash
>>
>> clang "$@"
>> - EOF -
>>
>> $ mycompiler --version
>> clang version 3.7.0 (tags/RELEASE_370/final)
>> Target: x86_64-unknown-linux-gnu
>> Thread model: posix
>>
>> Switching to GCC...
>>
>> [ /usr/bin/mycompiler ]
>>
>> #!/bin/bash
>>
>> gcc-4.9 "$@"
>> - EOF -
>>
>> $ mycompiler --version
>> gcc-4.9 (Ubuntu 4.9.2-0ubuntu1~12.04) 4.9.2
>> Copyright (C) 2014 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>
>> Thanks, that helped me a lot.
>>
>
> Sadly, this trick does not work here with Linux v4.2.
>

So, the problem is any change to your make-lines.
Means $COMPILER and/or compiler flags!

I simplified in Makefile...

COMPILER := clang
export COMPILER

and then did a symlink gcc-4.9 -> clang.

This did NOT work because clang uses a compiler-flag
'-no-integrated-as' which does not exist for gcc!

So, switching from gcc -> clang or vice-versa is not possible with
your compiler-wrapper-script trick :-(.

Anyway, I need a different solution.

One of my ideas was to hack the mm/Makefile.

--- a/mm/Makefile
+++ b/mm/Makefile
@@ -2,6 +2,9 @@
 # Makefile for the linux memory manager.
 #

+COMPILER := gcc
+export COMPILER
+
 KASAN_SANITIZE_slab_common.o := n
 KASAN_SANITIZE_slub.o := n

@@ -78,3 +81,6 @@ obj-$(CONFIG_CMA)     += cma.o
 obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o
 obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o
 obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o
+
+COMPILER := clang
+export COMPILER

Not sure if this works.

- Sedat -

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Tue, Sep 8, 2015 at 7:45 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> On Tue, Sep 8, 2015 at 12:30 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>> On Mon, Sep 7, 2015 at 11:20 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>> On Mon, Sep 7, 2015 at 10:59 PM, Linus Torvalds
>>> <torvalds@linux-foundation.org> wrote:
>>>> On Mon, Sep 7, 2015 at 1:53 PM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>>>>
>>>>> That does not work.
>>>>
>>>> .. because you didn't do what I told you to do.
>>>>
>>>>> I copied a gcc-compiled percpu.o OR deleted/renamed percpu.o and
>>>>> re-invoked make - this starts a complete new build from scratch.
>>>>
>>>> Right. Because you changed the compiler name, so now the build system
>>>> realizes that the old build instructions are stale.
>>>>
>>>> Which is why you have to:
>>>>
>>>>>> Use a wrapper around the compiler (and point to that wrapper with the
>>>>>> "to switch compilers from under the make, without the build paths
>>>>>> changing (because otherwise our makefile auto-machinery notices that
>>>>>> flags and command changed).
>>>>>>
>>>>>> Use CC (or CROSS_COMPILE) to point at your wrapper.
>>>>>
>>>>> No idea how to realize that, sorry.
>>>>
>>>> Literally just do something like this:
>>>>
>>>>  - have a shell script call "mycompiler" and make it do gcc/llvm "$@".
>>>>
>>>>  - or even just use a symlink (the script has the advantage that you
>>>> can play with the options etc too)
>>>>
>>>>  - change the shell script (or symlink) itself, and make sure to use
>>>> the same CC for "make" at all times, so that the build script never
>>>> sees that the underlying command is now different.
>>>>
>>>> It should work fine, I've done it a couple of times (although
>>>> admittedly not recently)
>>>>
>>>
>>> OK, I have created a mycompiler shell-script and use that for CC and
>>> HOSTCC in my own build-script.
>>>
>>> Using CLANG...
>>>
>>> [ /usr/bin/mycompiler ]
>>>
>>> #!/bin/bash
>>>
>>> clang "$@"
>>> - EOF -
>>>
>>> $ mycompiler --version
>>> clang version 3.7.0 (tags/RELEASE_370/final)
>>> Target: x86_64-unknown-linux-gnu
>>> Thread model: posix
>>>
>>> Switching to GCC...
>>>
>>> [ /usr/bin/mycompiler ]
>>>
>>> #!/bin/bash
>>>
>>> gcc-4.9 "$@"
>>> - EOF -
>>>
>>> $ mycompiler --version
>>> gcc-4.9 (Ubuntu 4.9.2-0ubuntu1~12.04) 4.9.2
>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>> This is free software; see the source for copying conditions.  There is NO
>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
>>>
>>> Thanks, that helped me a lot.
>>>
>>
>> Sadly, this trick does not work here with Linux v4.2.
>>
>
> So, the problem is any change to your make-lines.
> Means $COMPILER and/or compiler flags!
>
> I simplified in Makefile...
>
> COMPILER := clang
> export COMPILER
>
> and then did a symlink gcc-4.9 -> clang.
>
> This did NOT work because clang uses a compiler-flag
> '-no-integrated-as' which does not exist for gcc!
>
> So, switching from gcc -> clang or vice-versa is not possible with
> your compiler-wrapper-script trick :-(.
>
> Anyway, I need a different solution.
>
> One of my ideas was to hack the mm/Makefile.
>
> --- a/mm/Makefile
> +++ b/mm/Makefile
> @@ -2,6 +2,9 @@
>  # Makefile for the linux memory manager.
>  #
>
> +COMPILER := gcc
> +export COMPILER
> +
>  KASAN_SANITIZE_slab_common.o := n
>  KASAN_SANITIZE_slub.o := n
>
> @@ -78,3 +81,6 @@ obj-$(CONFIG_CMA)     += cma.o
>  obj-$(CONFIG_MEMORY_BALLOON) += balloon_compaction.o
>  obj-$(CONFIG_PAGE_EXTENSION) += page_ext.o
>  obj-$(CONFIG_CMA_DEBUGFS) += cma_debug.o
> +
> +COMPILER := clang
> +export COMPILER
>
> Not sure if this works.
>

This does not work.

And if it had worked - it cannot due to passing invalid compiler-flags to gcc.

Empty head.

- Sedat -

Re: Use (two) different compilers at build-time?

[Fengguang Wu]

On Mon, Sep 07, 2015 at 09:12:58PM +0200, Sedat Dilek wrote:
> Hi,
> 
> is it possible to use a different compiler at build-time?

btw, it'd be great if clang can just work on mainline kernel.

I tried to run clang in 0day kbuild tests, however make aborts
quickly in seconds. There are dozens of clang patches provided in

    http://llvm.linuxfoundation.org/index.php/Main_Page

however such our-of-tree patches are not bisect friendly.

Thanks,
Fengguang

Re: Use (two) different compilers at build-time?

[Sedat Dilek]

On Thu, Sep 10, 2015 at 2:25 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> On Mon, Sep 07, 2015 at 09:12:58PM +0200, Sedat Dilek wrote:
>> Hi,
>>
>> is it possible to use a different compiler at build-time?
>
> btw, it'd be great if clang can just work on mainline kernel.
>

I am not a member of that LLVMLinux team, but they upstreamed a lot of patches.

> I tried to run clang in 0day kbuild tests, however make aborts
> quickly in seconds. There are dozens of clang patches provided in
>
>     http://llvm.linuxfoundation.org/index.php/Main_Page
>
> however such our-of-tree patches are not bisect friendly.
>

I agree, there is a high potential for improvements in LLVMLinux :-).

Unfortunately, I struggled here some days with $COMPILER's
inline-optimization (disable | force| noinline) and it turned out to
be a "known" x86-hweight issue (a patch was archived and thrown away
from the series of x86 patches).

Finally, I have a "buildable" and "running-on-bare-metal"
llvmlinux-patched Linux v4.2 here on Ubuntu/precise AMD64.

It is everytime interesting to see where the root cause sits.
It's like in everyday life - never assume - just ask - stay curious
and attentive :-).

- Sedat -

[1] http://lists.linuxfoundation.org/pipermail/llvmlinux/2015-September/001357.html
[2] http://lists.linuxfoundation.org/pipermail/llvmlinux/2015-September/001358.html

Re: Use (two) different compilers at build-time?

[Fengguang Wu]

On Sun, Sep 13, 2015 at 05:21:41PM +0200, Sedat Dilek wrote:
> On Thu, Sep 10, 2015 at 2:25 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> > On Mon, Sep 07, 2015 at 09:12:58PM +0200, Sedat Dilek wrote:
> >> Hi,
> >>
> >> is it possible to use a different compiler at build-time?
> >
> > btw, it'd be great if clang can just work on mainline kernel.
> >
> 
> I am not a member of that LLVMLinux team, but they upstreamed a lot of patches.
> 
> > I tried to run clang in 0day kbuild tests, however make aborts
> > quickly in seconds. There are dozens of clang patches provided in
> >
> >     http://llvm.linuxfoundation.org/index.php/Main_Page
> >
> > however such our-of-tree patches are not bisect friendly.
> >
> 
> I agree, there is a high potential for improvements in LLVMLinux :-).
> 
> Unfortunately, I struggled here some days with $COMPILER's
> inline-optimization (disable | force| noinline) and it turned out to
> be a "known" x86-hweight issue (a patch was archived and thrown away
> from the series of x86 patches).
> 
> Finally, I have a "buildable" and "running-on-bare-metal"
> llvmlinux-patched Linux v4.2 here on Ubuntu/precise AMD64.

That's great!

> It is everytime interesting to see where the root cause sits.
> It's like in everyday life - never assume - just ask - stay curious
> and attentive :-).

If upstream kernels can just work (at least work for one kconfig),
I'll be able to follow up add run regular 0day build tests for LLVM.
Which should guarantee new break ups are detected ASAP and hopefully
the developer that breaks it may help fix it up "automatically". :-)

Thanks,
Fengguang