Cross-compiling C++11 without going mad(der)

on Friday, May 23, 2014

C++11 is all the rage these days. It’s got a ton of new features language- and compiler-wise that are aimed towards fixing the many problems that have constantly plagued C++ over the years. It’s not a perfect language, not by a long shot, it’s ridiculously verbose when following best practices and it’s, well, C++. I’m sure I’ll get flamed.



C++11 support


There’s one particular aspect of C++ that really appeals to me: it is the language that everyone* forgets is actually natively supported on the widest range of desktop and mobile platforms out there - OSX, Linux, Windows, Android, iOS, Raspberry, the newer consoles, etc, etc. And for the lazy programmer (that’s me) that wants to work on one platform and target all of the others without having to recode (much), C++ cross-compilation is tempting.


Now there’s various compilers out there with varying degrees of C++11 support. The one that annoys me the most is VisualStudio, since it’s been the slowest catching up and I would love to work on it. Alas, the only decent version that can actually compile most of the useful C++11 is VS2013, and I can’t even trust it to support proper defaulted constructors, initializer lists or move semantics. Aaaargh! Oh well, Windows is not a good platform for cross-compiling anyway, so let’s ignore VisualStudio and target Windows with GCC instead. Both Clang and GCC are considered done in terms of compiler features and between them, they cover the entire gamut of platforms I want to target (OSX, iOS and PS4 with clang, Linux, Windows and Android with GCC)


C++11 support is not only about the compiler features, but also about the standard library implementations. Clang ships libc++, GCC ships libstdc++. As of clang 3.4, libc++ is pretty much complete (although I hit a few bugs that I found fixed in libcxx/trunk). With gcc 4.8, libstdc++ is mostly complete.


To test compatibility and make sure that my C++11 code, built primarily on clang 3.4 on OSX, would compile and run on the other platforms, my first objective was to grab the libcxx tests, strip out the libcxxisms (like __has_feature() and LIBCPP defines) and cross compile them to run on ios (simulator), android, windows and linux. There’s a lot of tests and I wanted to go at it in stages, so I first tackled the threading tests, and also did some small threaded apps to further check how good the support is for std::future, std::thread, std::async and std::packaged_task. The results were very encouraging: all the code compiled and ran with no issues on osx, win (gcc and mingw x64), android (gcc armv7 and x86), ios (clang armv7 and sim/x86) and linux (gcc x64). Due to the limitations of the architecture, std::packaged_task and other features requiring atomics aren’t supported in armv6, so I decided to skip that arch.


I ran the algorithms tests, and found that the stdc++ shipped with gcc 4.8 doesn’t implement rotate. I get the feeling it’s not in 4.9 either, but I haven’t checked yet. Everything else from that suite built and ran fine.


In the atomics tests, things did not run quite so well. It looks like gcc doesn’t support the following syntax for initializing an atomic by passing a reference:


std::atomic_bool obj(true);
std::atomic_init(&obj, false);

Clang results vs GCC results


That breaks a bunch of tests. There’s other tests broken in that test suite too, another piece that gcc fails to compile is:


#include <atomic>

template <class T>
void test() {
    typedef std::atomic<T> athing;
    athing a;
}

struct A {
    int i;
    explicit A(int d = 0) : i(d) {}
};

int main() {
    test<A>();
}

It fails because the constructor of std::atomic is marked noexcept, but the test defines a struct A with an explicit constructor that defaults to no arguments (int d = 0) and it’s not marked as noexcept. GCC complains that the exception specification doesn’t match and fails to compile. Clang has no problems with it. This is one of those where I really am not sure which compiler is right (one works, one doesn’t, I’m tempted to say clang is more correct, but…).


There’s a bunch of other failures that I haven’t investigated yet, and a ton of tests I haven’t run, it’s something that’s going to take a while. This exercise has led me to believe that it’s very viable to cross-compile c++11 code, and by not relying on VS, I can use most features with no issues.


So how do I cross-compile?


I’m doing this on OSX (Mountain Lion) so I can target the widest range of platforms. Ideally, we’d use one compiler frontend and just switch the libraries around, but unfortunately this is not an ideal world, and using separate toolchains for every platform is safer and much easier.


Setup


If I can, I prefer to keep all toolchains in a directory called toolchains (obvious naming is obvious). Inside, android toolchains go into android/, windows into windows/, etc, etc. iOS toolchains are served from the system so you can symlink them in or just use the original paths.


Android


Android is pretty simple. Download the Android r9d NDK and dump it somewhere. I don’t use ndk-build directly if I can avoid it, I prefer to build native libraries with the standalone toolchain and later integrate them into apps using the include $(PREBUILT_SHARED_LIBRARY) mechanism that Android.mk files provide to link prebuilt libraries. To create standalone native android toolchains, you just run make-standalone-toolchain.sh:


$ANDROID_NDK_PATH/build/tools/make-standalone-toolchain.sh --platform=android-19 --install-dir=toolchains/android/arm --arch=arm --toolchain=arm-linux-androideabi-4.8

This will create an arm toolchain (suitable for all arm archs). For x86, use


--arch=x86 --toolchain=x86-4.8

There’s also a mips version:


--arch=mips --toolchain=mipsel-linux-android-4.8

Here’s an example command line for compiling with the arm toolchain:


$ANDROIDARM/bin/arm-linux-androideabi-g++  -Wall -std=c++11 -fno-rtti -g -O0 --sysroot=$ANDROIDARM/sysroot -march=armv7-a -MMD -MP -MF -fpic -ffunction-sections -funwind-tables -fstack-protector -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -pthread -DNDEBUG -c test.cpp -o test.o

And the corresponding link step:


$ANDROIDARM/bin/arm-linux-androideabi-g++ --sysroot=$ANDROIDARM/sysroot -no-canonical-prefixes -march=armv7-a -pthread -Wl,--fix-cortex-a8  -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -lstdc++ -lm  test.o -o test

You can run native apps built like this directly on an android device by copying them with adb to /data/local/tmp, which is nice for quick tests and automation that doesn’t require interaction with the Java runtime. For actual real android apps, you build an .so that you then either load from the Java side with LoadLibrary, load dynamically from a native activity (via ldopen, don’t forget to set the library path) or link statically to it. Android is getting good at exposing the system natively, so at this point there’s pratically no need for any Java code if the app does its own UI (like, say, a game).


iOS


Again, pretty simple stuff. You’ll need the SDK, of course, and for C++11 on OSX and iOS you really need a recent clang, so XCode 5 is required. You’ll be building for two targets, arm and the simulator (which is i386).


In toolchains/ios, do the following:


ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.1.sdk sim

ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.1.sdk arm

Note the version of the SDK in the path. Depending on what you have, you’ll need to adjust it.


Here’s an example command line for compiling for the simulator:


g++ -arch i386 -Wall -g -O0 -std=c++11 -stdlib=libc++ -fno-rtti --sysroot=$IOSSIM -D\_XOPEN\_SOURCE=1 -DTARGET\_IPHONE\_SIMULATOR -mios-simulator-version-min=5.0 -c test.cpp -o test.o

And linking:


g++ -arch i386   --sysroot=$IOSSIM -Wl,-syslibroot,$IOSSIM -stdlib=libc++ -mios-simulator-version-min=5.0 test.o -o test

And for iOS arm:


g++ -arch armv7 -Wall -g -O0 -std=c++11 -stdlib=libc++ -fno-rtti --sysroot=$IOSARM -DHAVE\_ARMV6=1 -DZ\_PREFIX -DPLATFORM\_IPHONE -DARM_FPU_VFP=1 -miphoneos-version-min=5.0 -mno-thumb -fvisibility=hidden -c test.cpp -o test.o

g++ -arch armv7  --sysroot=$IOSARM -Wl,-syslibroot,$IOSARM -stdlib=libc++ -miphoneos-version-min=5.0 test.o -o test

To use C++11, the minimum ios version is 5.0, so both command lines set that as a requirement.


$IOSSIM and $IOSARM point to the sim and arm symlinks created earlier. This will use the systems compiler, and only the location of target-specific libraries need to be specific (via sysroot).


The reason I’m dumping these command lines is so that it’s easier to see the parallels and automate them (in my case, via a small makefile).


Windows


Ah, Windows. Always the problematic little OS with the wonderful tools and the horrible build systems. Windows, of course, is not easy. Fortunately, it’s not that hard, either, because there’s something called MXE that’s going to make it a breeze to set up.


MXE (M cross environment) is a Makefile that compiles a cross compiler and cross compiles many free libraries such as SDL and Qt. Thus, it provides a nice cross compiling environment for various target platforms, which


It uses mingw32 and mingw64 and supports a ton of libraries. It’s pretty awesome.


Building the cross-compiler


Check out mxe


git clone -b stable https://github.com/mxe/mxe.git

The GCC version it’s set to build by default has a bug, so we need to change it to a newer one.


Edit src/gcc.mk and change the following lines to:


$(PKG)_VERSION  := 4.8.2
 $(PKG)_CHECKSUM := 810fb70bd721e1d9f446b6503afe0a9088b62986

Build a first version of the compiler and tools:


make MXE_TARGETS='x86_64-w64-mingw32' gcc gmp mpfr winpthreads lua -j4 JOBS=4

By default, gcc is configured to use win32 threads, which kills support for threading and other c++11 features, so after the first build of gcc is done, we’re going to build it again using pthreads.


Edit src/gcc.mk again and do the following:


On line 12, add winpthreads to the end of the $(PKG)_DEPS list so it looks like


$(PKG)_DEPS := mingwrt w32api mingw-w64 binutils gcc-gmp gcc-mpc gcc-mpfr winpthreads

On line 49, change --enable-threads=win32 to --enable-threads=posix


GCC isn’t configured to support sysroot by default, and it’s really handy to have when you’re cross-compiling, so we’re going to enable that.


Edit src/binutils.mk and add --with-sysroot to the list of flags, around line 38


Build gcc again


make MXE_TARGETS='x86_64-w64-mingw32' winpthreads gcc -j4 JOBS=4

Et voilá, a cross-compiler. You can now symlink the mxe/usr directory into toolchains/windows/x64 and use it like the android compiler.


Here’s an example command line for compiling for windows:


$WINDOWS64/bin/x86_64-w64-mingw32-g++ -Wall -g -O0 -std=c++11 -fno-rtti -pthread --sysroot=$WINDOWS64  -c test.cpp -o test.o

And linking:


$WINDOWS64/bin/x86_64-w64-mingw32-g++ --sysroot=$WINDOWS64 -lstdc++ -pthread test.o -o test.exe

Linux


Linux is GCC, of course, and there’s cross-compilers prebuilt for it. They’re annoying in that binutils wasn’t built with --enable-sysroot, which breaks my routine here a bit, but oh well.


You can find OSX-Linux cross-compilers for x86 and x64 at the crossgcc.rts site. They come in the form of dmg files, which I really don’t understand why. It’s a compiler, not an app, I need to set path flags anyway for it so I want to put it in a place of my choosing, not in the system. I don’t get it. Anyway, the dmg installs into /usr/local, so you can just copy them out from there after it’s done installing, or symlink them into your toolchains directory, like


ln -s /usr/local/gcc-4.8.1-for-linux64 toolchains/linux/x64

Here’s an example command line for compiling for linux:


$LINUX64/bin/x86_64-pc-linux-g++ -Wall -g -O0 -std=c++11 -fno-rtti -pthread --sysroot=$LINUX64 -c test.cpp -o test.o

And linking:


$LINUX64/bin/x86_64-pc-linux-g++ -L$LINUX64 -L$LINUX64/lib64 -static-libstdc++ -pthread test.o -o test

Note the lack of --sysroot during linking. Also note the -static-libstdc++ flag. This flag will ensure that libstdc++ will be linked statically, which will ensure that your app actually runs on whatever linux system you’re going to try to run it on. libstdc++ changes often, and the linking will link a specific version in, which may or may not be found on the target system, with amusing results


Epilogue


Phew! If you got this far, congratulations. This mostly served as a dumping ground for stuff I don’t want to forget, so your mileage may vary. Now go and do some cross-compilation, I’ve got a game engine to write.




* not everyone

9 comments:

Chris Toshok said...

just a short comment as I read: I run VS2013 on win7.

andreia|gaita said...

Ah, it must have complained at me because of some missing service pack and I assumed it was the win version. Post fixed, thanks :)

Paco Martinez said...
This comment has been removed by the author.
Paco Martinez said...

Andreia, great read. Thank you for your work. You have now reignited my interest in Cpp.

Hope everything is going well for you!

Keep up the great work you do.

Paco

amartins said...

Nice post! But why would you want to write yet another game engine? :)

Chetan said...

Extremely useful nuggets of information. Thanks for spending the time to put together this post.

Andreia Gaita said...

@Paco Thanks! C++11 is a really interesting new iteration on C++. You can finally do block locks like in C# for thread-safe code:

lock (mutex, [&] {
std::cout << "Processing " << filename << "\n";
});

Yes, the syntax is weird, but I have to commend the effort, they're really trying to improve it.

Andreia Gaita said...

@amartins The question is, why not? ;)

Seriously though, a game engine is an excellent playground for training and experimenting with all sorts of things.

It needs to be fast, so you have to think about cache lines and data alignment, which forces you out of the traditional OO design mentality and into actually examining design decisions very carefully and making compromises. It needs threading, so it probably needs a dispatching system of some sort, that comes all kinds of multithreaded fun.

You're not bound by the UI paradigms of the OS, because you do your own UI, so that makes you think about how to actually do a UI. That takes you deep into opengl and shader territory and the differences between implementations available on desktop and mobile, and then you start thinking "maybe I could do a UI toolkit and build from it" which takes you into all kinds of interesting design decisions.

Being cross-platform requires you to investigate compilers and build systems and the best way to share code across platforms minimizing effort, while not having to worry about porting libraries because it's a game engine, you're not going to be tying it into OS systems very much so it doesn't pull in libraries that need porting (and the ones you do use, like sdl or fmod, are already ported)

It needs a scripting environment for ease of use, so that requires you to not only pick a language, but build an environment that it can interact with, and that requires a clean API. There's a ton of design decisions to be made here: how much power do you give the user? how do you expose your api from the native side (especially given that it's c++)? You could auto-generate an api layer, or you could do it manually. And choice of language, functional, imperative, does it come with its own libraries or do you provide everything (io access, threading, networking, etc, etc)?

And then, of course, you can make a game. Eventually :D

amartins said...

Hehe Andreia. Fabulastic answer! You see, knowing all those points was what made me ask you the question, most people i find aren't aware of them. ;)
The amount of planning, work and energy needed to create anything like a usable multi-platform game engine, is just like you perfectly illustrated, daunting..
On the other hand, yes, it's without any doubt an amazing opportunity to experiment and train on new things such as languages or paradigms.
Me and myself have been wanting to go down that road for quite some time now. Actually what took me to look at this post was exactly that interest. I have been hoarding such resources for a long time now. :/
Anyway, maybe we can exchange some ideas if move forward with your project..

Cheers,