Have seen a couple interesting articles[0][1] highlighting the weirdness of building Chromium on Windows, so hopefully this LLVM improvement will be a step towards a smoother process.
Hey, those are mine! Those two issues (and this one: https://randomascii.wordpress.com/2018/02/11/zombie-processe...) are equally problematic with clang. Two of them were problems with Windows, and one was a flaw in a Microsoft system management component.
Luckily they are all mitigated, fixed, or being fixed, so my Chrome builds run much more smoothly now.
> if your project is Windows-only, you can get a second compiler’s opinion on your code, and Clang’s warnings might find bugs
GCC (mingw.org and mingw-w64 triples) has existed and been able to do this, including and especially cross-compiling, for a long time. No need to manually download a Windows SDK either.
edited to add: to be fair, it's not ABI compatible with MSVC. But it deserves a mention and it's an easier option than MSVC-style build systems for many open source projects.
Over the past 2-3 years, with great help from a few other LLVM devs I have slowly pushed for native clang support for mingw-w64. With the current in tree HEAD you can now build and bootstrap mingw-w64 with a llvm-only toolchain (no binutils or gcc, thus no disregard of the PECOFF SPEC)
The stack is as follows. LLVM+CLANG+LLD+COMPILER-RT+LIBCXX+LIBCXXABI+LIBUNWIND+MINGW-W64.
The libraries and executables built this way can be dropped into visual studio quite easily, the two environments are now basically interchangeable. This also means we can borrow the PDB debugging work by Zachary Turner and/or any msvc features added to llvm for free that was done by the various googlers working on chrome. See a talk from Reid on the most recent LLVM developer meetup for more info on PDB and how it works.
There has been no formal release of this environment yet but you should see something in the coming weeks/months.
I keep build scripts here for those interested.
I'm due another cleanup of the repo now that wine 3.0 is out so I can actually run x64 tests within a linux docker container.
I'm going to be doing some blogging to make this more visible with various partners that want to support this.
The most interesting part of this for me was implementing a llvm-dlltool alternative to binutils dlltool that actually works within the PECOFF spec so msvc lib.exe and link.exe likes what it produces.
This sounds great, it's been a long time coming and I'm sure many people will appreciate being able to debug their mingw generated executables in Visual Studio.
I did all of the PDB stuff in LLVM, happy to help if you need it (ping me on the mailing list or IRC)
Excellent work. I like to ignore visual studio completely and pretend it doesn't exist, but that's just me :)
I've used cygwin's cross-compiler x86_64-w64-mingw32-clang package once in a while, though I find more projects' build systems choke on it and do the wrong thing (libgmp as a notable example) than with x86_64-w64-mingw32-gcc.
I tend to ignore both gcc and MSVC. Clang is really good at bootstrapping itself with this setup even when cross compiling without any real complexity. Without diverting into a cmake vs autotools battle I like that llvm and clang just work and bootstrap so easily. Gone are the days of messing around with gmp isl and co to bootstrap a toolchain.
Fair enough, but I actually need gmp more for runtime math, plus fortran deps like lapack and more obscure things. Maybe flang will be production-worthy some day - do you know if they're still outputting a textual representation of IR into string buffers?
I've been watching the process of getting git to build with VS, but it's sounding like mingw-w64 will gain PDB output faster than the VS version will happen. Either way, if I can debug (and profile) git.exe then I'm happy.
Great work! We with Nixpkgs/NixOS have over the past year overhauled cross compilation, culminating recently with a bunch of parallel efforts recently including Windows cross compilation. People have built Nix with cygwin in the past, it would be nice to get it working in MSYS2/mingw-w64 too.
One of mingw{,-w64}'s major contributions is a gcc-compatible replacement set of headers and import libraries, instead of using the Windows SDK or MSVC versions.
These work great most of the time, but if you start hooking deeply into windows, you realise they are a little bit outdated or incompatible. e.g. I get a lot of errors like
#warning COM interfaces layout in this header has not been verified.
#warning COM interfaces with incorrect layout may not work at all.
#pragma message: Interface Ifoo has unverified layout.
in COM code, and some ATL things it is unable to build at all.
Apple hired developers of Clang and started supporting Clang around the time GPLv3 was being pushed by FSF, Apple at the time was using GCC in their development and feared that GPLv3 would taint their code.
other than technical merits, permissible license of Clang is attractive for some developers/businesses
> edited to add: to be fair, it's not ABI compatible with MSVC. But it deserves a mention and it's an easier option than MSVC-style build systems for many open source projects.
A lot of newer Win32 APIs are IUnknown-based COM interfaces. APIs that are probably relevant for Chrome are shell interop, drag & drop, Direct2D, Direct Write, Media Foundation.
Does this option give you full debug information? My impression was that it does not. I require full debug information for debugging and profiling, and with clang-cl I get that (context: I work on Chromium)
> On the other hand, if all platforms use the same compiler, if it builds on your machine then it’s probably going to build on all platforms.
This also works the other way: buggy code that "works" in Clang may now go unnoticed.
This means it will be harder to build Chromium using GCC or MSVC. I see that Arch Linux already has to depend on Clang to build Chromium [1], which isn't very reassuring.
Yeah, just things that aren't optimized yet.
Windows headers, etc have very different hot spots and need different paths to be fast than linux/mac ones.
For reference, in distributed builds, the build times are ridiculously faster using clang.
(it shaves many hours off the time).
It's literally things like "parsing large numbers of pragmas is slower in clang because nobody has bothered to optimize it".
This is not uncommon for very different codebases.
Clang was, for a long time, many times slower at compiling the linux kernel, for example, because the linux kernel headers have a huge amount of inline assembly and nobody ever bothered to optimize the parsing paths for inline assembly.
(now somewhat fixed).
This is just the very typical "hey, when you change the use cases dramatically, you may need to retune/rethink how certain things are done"
I think people just may not realize how wildly different these types of codebases are in what types of statements/things occur.
clang isn't. clang-cl is (after years of work by the clang-cl team at Google). They had to do a lot of work, especially around PDB (symbol-file) generation.
clang-cl is a drop-in replacement for cl (even the command lines are the same).
But clang-cl is just a driver for clang, and it's still possible to get the ABI compatibility without clang-cl, but you have to know what you're doing and get the flags right.
So, as the person responsible for deciding to make just about all of the compiler switches happen (and i used to run opensource licensing for Google), this is just coincidence[1].
In this case, we just felt clang/llvm was a better architecture to build on at this point. These are also many-year projects.
I was a GCC/gdb/etc maintainer for many years, so i've got nothing against GPL software.
[1] Well, part of it is that by numbers, most newer open source projects are not GPL. The percent of GPL'd new projects before, say, 2010 and after 2010 is very different. I haven't gone far back to see when that starts, but the percentages are quite significant. It used to be 40-50% of new projects, and now it's like 10-20, maybe.
Thank you for taking the time to shed some light on this. I understand your point about a multi-year project. I actually worked on a debugger before (to support a proprietary complier) so I know decisions in that area are far from random.
As a note for myself, I should check out llvm in greater detail.
I doubt this is very relevant in the case of Chrome compilers. It's more about the huge investment/expertise in Clang- or LLVM-based dynamic analysis and development tooling.
It's just the better platform for writing these things. Heck, this is what is was built for in the first place. GCC is a solid, but also a very traditional compiler.
> Heck, this is what is was built for in the first place.
LLVM was supposed to be a virtual machine environment (hence the name); I remember when it came out and I was in college doing program language research. It entirely dropped that history and got twisted into a compiler platform much later. And clang only came into existence because the developers of g c (in particular, Richard Stallman) did not want to work with Apple on maintaining LLVM as a gdc backend (which was a stupid stupid decision: Apple said they would be willing to give the copyright of LLVM to the FSF, which would have kept all compiler tech under GPL), which was all well after Apple had been switching to LLVM for their compiler stack (in order to obtain weird optimizations).
First:
We were already publishing all our GCC changes, as we did all our work upstream (we worked on a staging branch in the gcc repo, and changes were moved to trunk as fast as possible) Plenty of other companies used our branch and we were happy with it.
We do all our work upstream for clang/llvm as well, and pull compiler releases when upstream is green according to our internal testing.
(Pretty much the only cherry picks are a small number of reverts sometimes)
Second, literally nobody has ever come and expressed a concern about using GCC to me.
I've had to explain the runtime and libstdc++ exceptions to people, but that was the other way around -- They weren't worrying, and I needed to make sure they worried and shipped source when they ship it as a .so.
That's true for any developer of embedded software. Any one who makes hardware of any kind will avoid trying to put any GPLv3 licensed code on their hardware.
With Microsoft producing MSVC/C2 compiler with Clang frontend, I am surprised they did not abandon the MSVC at all - they have a hard times even at supporting C99, not to mention modern C++ technologies. And all those tools and plugins built around Clang/LLVM. MSVC is a legacy of XX century and should be abandoned for good.
All the updates regarding C99 and C11 in Visual C++ are related to ANSI/ISO C++ requirements regarding C compatibility, as Herb states.
Regarding legacy, Visual C++ is one of the top C++ compilers doing incremental compilation and linking, which still beats Rust compilation times on Windows.
C99 and C11 library support is required by ANSI/ISO C++.
As for the minor language changes, there was a blog post that I cannot locate now, stating it was to the extent required for porting well known projects to Windows, the remaining features would depend on key customers feedback.
> They don't have any hard time supporting C99, from Microsoft point of view C is done, time to move on to C++.
If that's the case then Microsoft should get its corporate ass out of C's standardization committee. It makes no sense to waste their time forcing requirements onto a standard that they refuse to comply with.
I’m sure you won’t like this comment, but doesn’t Microsoft have a vested interest in keeping C as unchanged as possible? If C were to run away in unorthodox directions, that would hurt compatibility with C++ and hence Visual Studio. So from their perspective it may make perfect sense to stay on the committee to keep it as conservative as possible. Why voluntarily give up soft power?
There's a link in the article here, but sadly it sounds like they've abandoned Clang/C2. I guess they got normal MSVC up to speed on the C++1y/z they wanted to support, and Clang/LLVM was always better than both at complex numbers, inline assembly, and other C features.
Who are the "we" they refer to in this blog? Are they Google's devs who build chrome, or are they clang's devs? If they are clang's dev, why are they building chrome? I am completely confused on this one.
1) ABI compatibility is not present. With clang-cl you can compile half of your object files with MSVC and half with clang-cl, and link them together. That's very powerful. You might be thinking "but who is really gonna do that anyway?" However, there is more to the ABI than just this. For example, consider what happens if you need to use Structured Exception Handling. Well, with GCC, you can't.
2) Debug Information. GCC does not emit PDBs, it emits DWARF. This means you cannot use Visual Studio to debug, you cannot use vTune or WPA to profile, you cannot use symbol / source servers.
3) There are licensing implications, GPL != BSD and it's easier to use BSD.
These are the main reasons. There are other reasons though, such as if the entire goal is to get down to 1 toolchain and that toolchain is X, then why would you waste your time converting Y to Z?
At one point, after updating the toolchain from gcc 4.x to 7.x and from mingw32 to mingw-w64, it started to produce broken binaries.
The manifestation of that brokenness was that binaries refused to start properly on systems where the toolchain was not installed. It just silently exited during startup - no crash, or error message, or anything.
This applied both to binaries that were cross-built on linux, and binaries built using MSYS2 on Windows.
I decided to just switch to clang-cl + MSVC headers instead, which worked properly. The only gotcha there was the need to store the headers on a case insensitive file system (I chose vfat) when compiling on linux, because the case of the includes aren't consistent (e.g. Windows.h vs. windows.h), even in the MSVC headers themselves.
were you missing compiler runtime libraries? like any code compiled with gcc anywhere that links the runtimes as shared libraries, they'll need to be present for executables to run. depending which variant of mingw-w64 you used, there are more runtime library dependencies than there used to be, like winpthreads for c++11 thread support.
Last time I tried to build Chrome/Chromium I had to mess about with this build system called Ninja. (Which worked well it was just awkward to set up on Windows). Is that still in play?
It's by design that you need a meta-build system. Ninja build files were not designed to be written by hand, not even ninja itself has hand written ninja files.
My point was that, in the context of Chrome using ninja, the claim that "Ninja is used because it's simple, multiplatform and very fast. The mainstream alternatives offer at most 2 out of 3." is a bit odd considering that the decision doesn't seem to be ninja vs mainstream alternatives but rather ninja+gn(or gyp or whatever else) vs alternatives.
Can you elaborate on the difficulty on Windows? If you go to the home page there is a section "Getting Ninja" that links to a page with a zip file... ?
Did the LLVM folks fix the C++ structured exception handling incompatibility between MSVC compiled DLLs/programs and LLVM compiled ones? This was a problem for many years.
Tried to install Clang 5.0 Visual Studio integration, unfortunately it seems that I need to install VS 2015 (v140) compiler since it doesn't work with VS 2017 (v141).
Asynchronous Exceptions (SEH): Partial. Structured exceptions (__try / __except / __finally) mostly work on x86 and x64. LLVM does not model asynchronous exceptions, so it is currently impossible to catch an asynchronous exception generated in the same frame as the catching __try.
I've tried clang for windows twice. Each time the problem hasn't been the compiler, which works well, it has been which library to use with it on windows. Clang 5.0 worked well up until I needed the filesystem library, which isn't there. Does anyone know if it still relies on the visual studio C++ library?
clang-cl is just a compiler. It relies on VS to provide the C and C++ standard libraries. VS 2017 has the filesystem library and it should work with clang-cl 6.
To elaborate on that, if you want to build Chromium for Windows you currently still need to install Visual Studio. We don't use its compiler anymore but we use its linker and other tools, its header files, and its libraries. Many Chromium developers also use it for editing code, and for debugging code.
Chrome is currently built using VS 2017 15.3 (that's update 3). Eventually we will probably upgrade to VS 2017 15.7 in order to get more STL conformance, linker improvements, etc.
[0] https://randomascii.wordpress.com/2018/02/25/compiler-bug-li...
[1] https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...