Cross-compiling C++11 without going mad(der)

Publish date: May 23, 2014
Tags: c++

Table of contents

  1. Setup
  2. Android
  3. iOS
  4. Windows

C++11 is all the rage these days. It’s got a ton of new features language- and compiler-wise that are aimed towards fixing the many problems that have constantly plagued C++ over the years. It’s not a perfect language, not by a long shot, it’s ridiculously verbose when following best practices and it’s, well, C++. I’m sure I’ll get flamed.

C++11 support

There’s one particular aspect of C++ that really appeals to me: it is the language that everyone* forgets is actually natively supported on the widest range of desktop and mobile platforms out there - OSX, Linux, Windows, Android, iOS, Raspberry, the newer consoles, etc, etc. And for the lazy programmer (that’s me) that wants to work on one platform and target all of the others without having to recode (much), C++ cross-compilation is tempting.

Now there’s various compilers out there with varying degrees of C++11 support. The one that annoys me the most is VisualStudio, since it’s been the slowest catching up and I would love to work on it. Alas, the only decent version that can actually compile most of the useful C++11 is VS2013, and I can’t even trust it to support proper defaulted constructors, initializer lists or move semantics. Aaaargh! Oh well, Windows is not a good platform for cross-compiling anyway, so let’s ignore VisualStudio and target Windows with GCC instead. Both Clang and GCC are considered done in terms of compiler features and between them, they cover the entire gamut of platforms I want to target (OSX, iOS and PS4 with clang, Linux, Windows and Android with GCC)

C++11 support is not only about the compiler features, but also about the standard library implementations. Clang ships libc++, GCC ships libstdc++. As of clang 3.4, libc++ is pretty much complete (although I hit a few bugs that I found fixed in libcxx/trunk). With gcc 4.8, libstdc++ is mostly complete.

To test compatibility and make sure that my C++11 code, built primarily on clang 3.4 on OSX, would compile and run on the other platforms, my first objective was to grab the libcxx tests, strip out the libcxxisms (like __has_feature() and LIBCPP defines) and cross compile them to run on ios (simulator), android, windows and linux. There’s a lot of tests and I wanted to go at it in stages, so I first tackled the threading tests, and also did some small threaded apps to further check how good the support is for std::future, std::thread, std::async and std::packaged_task. The results were very encouraging: all the code compiled and ran with no issues on osx, win (gcc and mingw x64), android (gcc armv7 and x86), ios (clang armv7 and sim/x86) and linux (gcc x64). Due to the limitations of the architecture, std::packaged_task and other features requiring atomics aren’t supported in armv6, so I decided to skip that arch.

I ran the algorithms tests, and found that the stdc++ shipped with gcc 4.8 doesn’t implement rotate. I get the feeling it’s not in 4.9 either, but I haven’t checked yet. Everything else from that suite built and ran fine.

In the atomics tests, things did not run quite so well. It looks like gcc doesn’t support the following syntax for initializing an atomic by passing a reference:

std::atomic_bool obj(true);
std::atomic_init(&obj, false);

Clang results vs GCC results

That breaks a bunch of tests. There’s other tests broken in that test suite too, another piece that gcc fails to compile is:

#include <atomic>

template <class T>
void test() {
    typedef std::atomic<T> athing;
    athing a;
}

struct A {
    int i;
    explicit A(int d = 0) : i(d) {}
};

int main() {
    test<A>();
}

It fails because the constructor of std::atomic is marked noexcept, but the test defines a struct A with an explicit constructor that defaults to no arguments (int d = 0) and it’s not marked as noexcept. GCC complains that the exception specification doesn’t match and fails to compile. Clang has no problems with it. This is one of those where I really am not sure which compiler is right (one works, one doesn’t, I’m tempted to say clang is more correct, but…).

There’s a bunch of other failures that I haven’t investigated yet, and a ton of tests I haven’t run, it’s something that’s going to take a while. This exercise has led me to believe that it’s very viable to cross-compile c++11 code, and by not relying on VS, I can use most features with no issues.

So how do I cross-compile?

I’m doing this on OSX (Mountain Lion) so I can target the widest range of platforms. Ideally, we’d use one compiler frontend and just switch the libraries around, but unfortunately this is not an ideal world, and using separate toolchains for every platform is safer and much easier.

Setup

If I can, I prefer to keep all toolchains in a directory called toolchains (obvious naming is obvious). Inside, android toolchains go into android/, windows into windows/, etc, etc. iOS toolchains are served from the system so you can symlink them in or just use the original paths.

Android

Android is pretty simple. Download the Android r9d NDK and dump it somewhere. I don’t use ndk-build directly if I can avoid it, I prefer to build native libraries with the standalone toolchain and later integrate them into apps using the include $(PREBUILT_SHARED_LIBRARY) mechanism that Android.mk files provide to link prebuilt libraries. To create standalone native android toolchains, you just run make-standalone-toolchain.sh:

$ANDROID_NDK_PATH/build/tools/make-standalone-toolchain.sh --platform=android-19 --install-dir=toolchains/android/arm --arch=arm --toolchain=arm-linux-androideabi-4.8

This will create an arm toolchain (suitable for all arm archs). For x86, use

--arch=x86 --toolchain=x86-4.8

There’s also a mips version:

--arch=mips --toolchain=mipsel-linux-android-4.8

Here’s an example command line for compiling with the arm toolchain:

$ANDROIDARM/bin/arm-linux-androideabi-g++  -Wall -std=c++11 -fno-rtti -g -O0 --sysroot=$ANDROIDARM/sysroot -march=armv7-a -MMD -MP -MF -fpic -ffunction-sections -funwind-tables -fstack-protector -mfpu=vfpv3-d16 -mfloat-abi=softfp -mthumb -fomit-frame-pointer -fno-strict-aliasing -finline-limit=64 -pthread -DNDEBUG -c test.cpp -o test.o

And the corresponding link step:

$ANDROIDARM/bin/arm-linux-androideabi-g++ --sysroot=$ANDROIDARM/sysroot -no-canonical-prefixes -march=armv7-a -pthread -Wl,--fix-cortex-a8  -Wl,--no-undefined -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -lstdc++ -lm  test.o -o test

You can run native apps built like this directly on an android device by copying them with adb to /data/local/tmp, which is nice for quick tests and automation that doesn’t require interaction with the Java runtime. For actual real android apps, you build an .so that you then either load from the Java side with LoadLibrary, load dynamically from a native activity (via ldopen, don’t forget to set the library path) or link statically to it. Android is getting good at exposing the system natively, so at this point there’s pratically no need for any Java code if the app does its own UI (like, say, a game).

iOS

Again, pretty simple stuff. You’ll need the SDK, of course, and for C++11 on OSX and iOS you really need a recent clang, so XCode 5 is required. You’ll be building for two targets, arm and the simulator (which is i386).

In toolchains/ios, do the following:

ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator7.1.sdk sim

ln -s /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.1.sdk arm

Note the version of the SDK in the path. Depending on what you have, you’ll need to adjust it.

Here’s an example command line for compiling for the simulator:

g++ -arch i386 -Wall -g -O0 -std=c++11 -stdlib=libc++ -fno-rtti --sysroot=$IOSSIM -D_XOPEN_SOURCE=1 -DTARGET_IPHONE_SIMULATOR -mios-simulator-version-min=5.0 -c test.cpp -o test.o

And linking:

g++ -arch i386   --sysroot=$IOSSIM -Wl,-syslibroot,$IOSSIM -stdlib=libc++ -mios-simulator-version-min=5.0 test.o -o test

And for iOS arm:

g++ -arch armv7 -Wall -g -O0 -std=c++11 -stdlib=libc++ -fno-rtti --sysroot=$IOSARM -DHAVE_ARMV6=1 -DZ_PREFIX -DPLATFORM_IPHONE -DARM_FPU_VFP=1 -miphoneos-version-min=5.0 -mno-thumb -fvisibility=hidden -c test.cpp -o test.o

g++ -arch armv7  --sysroot=$IOSARM -Wl,-syslibroot,$IOSARM -stdlib=libc++ -miphoneos-version-min=5.0 test.o -o test

To use C++11, the minimum ios version is 5.0, so both command lines set that as a requirement.

$IOSSIM and $IOSARM point to the sim and arm symlinks created earlier. This will use the systems compiler, and only the location of target-specific libraries need to be specific (via sysroot).

The reason I’m dumping these command lines is so that it’s easier to see the parallels and automate them (in my case, via a small makefile).

Windows

Ah, Windows. Always the problematic little OS with the wonderful tools and the horrible build systems. Windows, of course, is not easy. Fortunately, it’s not that hard, either, because there’s something called MXE that’s going to make it a breeze to set up.

MXE (M cross environment) is a Makefile that compiles a cross compiler and cross compiles many free libraries such as SDL and Qt. Thus, it provides a nice cross compiling environment for various target platforms, which

It uses mingw32 and mingw64 and supports a ton of libraries. It’s pretty awesome.

Building the cross-compiler

Check out mxe

git clone -b stable https://github.com/mxe/mxe.git

The GCC version it’s set to build by default has a bug, so we need to change it to a newer one.

Edit src/gcc.mk and change the following lines to:

$(PKG)_VERSION  := 4.8.2
 $(PKG)_CHECKSUM := 810fb70bd721e1d9f446b6503afe0a9088b62986

Build a first version of the compiler and tools:

make MXE_TARGETS='x86_64-w64-mingw32' gcc gmp mpfr winpthreads lua -j4 JOBS=4

By default, gcc is configured to use win32 threads, which kills support for threading and other c++11 features, so after the first build of gcc is done, we’re going to build it again using pthreads.

Edit src/gcc.mk again and do the following:

On line 12, add winpthreads to the end of the $(PKG)_DEPS list so it looks like

$(PKG)_DEPS := mingwrt w32api mingw-w64 binutils gcc-gmp gcc-mpc gcc-mpfr winpthreads

On line 49, change --enable-threads=win32 to --enable-threads=posix

GCC isn’t configured to support sysroot by default, and it’s really handy to have when you’re cross-compiling, so we’re going to enable that.

Edit src/binutils.mk and add --with-sysroot to the list of flags, around line 38

Build gcc again

make MXE_TARGETS='x86_64-w64-mingw32' winpthreads gcc -j4 JOBS=4

Et voilá, a cross-compiler. You can now symlink the mxe/usr directory into toolchains/windows/x64 and use it like the android compiler.

Here’s an example command line for compiling for windows:

$WINDOWS64/bin/x86_64-w64-mingw32-g++ -Wall -g -O0 -std=c++11 -fno-rtti -pthread --sysroot=$WINDOWS64  -c test.cpp -o test.o

And linking:

$WINDOWS64/bin/x86_64-w64-mingw32-g++ --sysroot=$WINDOWS64 -lstdc++ -pthread test.o -o test.exe
``

## Linux

Linux is GCC, of course, and there’s cross-compilers prebuilt for it. They’re annoying in that binutils wasn’t built with `--enable-sysroot`, which breaks my routine here a bit, but oh well.

You can find OSX-Linux cross-compilers for x86 and x64 at the [crossgcc.rts site](http://crossgcc.rts-software.org/doku.php?id=compiling_for_linux). They come in the form of dmg files, which I really don’t understand why. It’s a compiler, not an app, I need to set path flags anyway for it so I want to put it in a place of my choosing, not in the system. I don’t get it. Anyway, the dmg installs into /usr/local, so you can just copy them out from there after it’s done installing, or symlink them into your toolchains directory, like

ln -s /usr/local/gcc-4.8.1-for-linux64 toolchains/linux/x64


Here’s an example command line for compiling for linux:

$LINUX64/bin/x86_64-pc-linux-g++ -Wall -g -O0 -std=c++11 -fno-rtti -pthread –sysroot=$LINUX64 -c test.cpp -o test.o


And linking:

$LINUX64/bin/x86_64-pc-linux-g++ -L$LINUX64 -L$LINUX64/lib64 -static-libstdc++ -pthread test.o -o test

Note the lack of --sysroot during linking. Also note the -static-libstdc++ flag. This flag will ensure that libstdc++ will be linked statically, which will ensure that your app actually runs on whatever linux system you’re going to try to run it on. libstdc++ changes often, and the linking will link a specific version in, which may or may not be found on the target system, with amusing results

Epilogue

Phew! If you got this far, congratulations. This mostly served as a dumping ground for stuff I don’t want to forget, so your mileage may vary. Now go and do some cross-compilation, I’ve got a game engine to write.