Statically link to C++ standard library

[daira]

Dynamically linking to these libraries [edit: before #4060, this referred to libstdc++ and libgomp] has in practice caused too many problems for users (#2358, #2257, #1840, #1692, #2468). In principle, it should be possible to just declare a dependency on the right version according to which g++ major version the code was compiled with, and the OS package management should sort it out. But in practice that does not work because commonly used distributions do not provide backports of the necessary packages in a timely fashion (or that can actually be installed without causing conflicts), for the distribution versions that many users are still using and that are supposed to still be supported.

Note that libc, libpthreads and libm are designed to be dynamically linked; we should not change the linking of these. libgcc (the support library for arithmetic primitives and exception handling) is a tricky case; I'm unsure whether to make this statically or dynamically linked given that libstdc++ is static (you can find articles on the web advocating either approach, and neither seems to be without difficulty).

Edit, 2020-08-03: This ticket now refers to statically linking libc++ using Clang. See also #4613 (comment)

[daira]

When building from source, the default should continue to be to dynamically link to these libraries, because if you have a given version of gcc/g++ then you should also have the runtime libraries associated with it. (If you're cross-compiling then it's up to you to make sure the target system has them.)

[daira]

http://micro.nicholaswilson.me.uk/post/31855915892/rules-of-static-linking-libstdc-libc-libgcc says "Statically linking to libgcc: Depends on the system. Never ever do this on a GNU-based system (ie a system that ships with libgcc or gcc, or where libc was compiled with gcc, or where libc is glibc, etc)."

Therefore I think we want --static-libstdc++, but not --static-libgcc, and to explicitly link with libgomp.a.

[daira]

Note that we MUST make sure that the resulting binary does not link both dynamically and statically to either of these libraries, because disaster will ensue at runtime if that happens. The approach can only work because we're linking statically to all the other libraries that use C++ or OpenMP.

[daira]

--static-libstdc++ apparently forces static linking of libgomp (when -fopenmp is used) according to https://gcc.gnu.org/bugzilla/show_bug.cgi?id=41605 .

[daira]

Also note that --static-libstdc++ might not actually be sufficient to produce an executable that works on some systems with older versions of the system dynamic linker: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60348 (which is a perfectly valid bug despite the resolution and the unhelpful attitude of gcc maintainers displayed there). But that's technically a separate problem even though it manifests, confusingly, as missing libstdc++ symbols at run-time.

[daira]

Here is the argument gcc developers originally made for libgcc being provided as a dynamic library and linked that way by default: https://gcc.gnu.org/ml/gcc/2000-04/msg00610.html . Note that it's essentially the same argument as for making libstdc++ linked dynamically by default, except that in practice we have the following differences between the two libraries:

• libgcc is stable; libstdc++ is not. It is rare for any new gcc feature or standards conformance improvement to require a new libgcc.
• libgcc has a mostly pure procedural API. By that I mean that the math support functions are pure-functional, and the exception-handling functions, although they do have control side-effects, don't use abstract data types that might change representation.
• if the ABI of libgcc changed, it would probably have to be done by adding new functions with different entry points rather than changing the behaviour of the old functions.

Therefore I believe it's safe to dynamically link against libgcc but statically link against libstdc++ (and libgomp), and that doing so will cause the fewest compatibility problems for the Gitian builds.

[daira]

This appears to work:

daira@sappho:~/zecc/zcash$ cat omp.cpp
#include <omp.h>
#include <iostream>

int main(int argc, char **argv) {
    std::cout << omp_get_num_threads() << std::endl;
    return 0;
}
daira@sappho:~/zecc/zcash$ g++ --version
g++ (Debian 5.3.1-8) 5.3.1 20160205
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

daira@sappho:~/zecc/zcash$ g++ -static-libstdc++ -fopenmp -c omp.cpp -o omp.o
daira@sappho:~/zecc/zcash$ g++ -static-libstdc++ omp.o $(g++ --print-file-name=libgomp.a) \
    @-ldl -pthread -o omp
daira@sappho:~/zecc/zcash$ ./omp
1
daira@sappho:~/zecc/zcash$ ldd omp
	linux-vdso.so.1 (0x00007ffd9876e000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f57fced2000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f57fcbcd000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f57fc9b5000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f57fc798000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f57fc3f7000)
	/lib64/ld-linux-x86-64.so.2 (0x000055dde9083000)

However, I had some difficulty getting the Zcash build process to use these flags (it was silently dynamically linking to libstdc++ and libgomp despite the -static-libstdc++ flag). I'll try again later, or perhaps someone can pair with me on this.

[daira]

Here's the error I get:

libtool: link: /home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/share/../native/bin/ccache g++ -m64 -std=c++11 -pipe -O1 -fwrapv -fno-strict-aliasing -Werror -g -static-libstdc++ -Wformat -Wformat-security -Wstack-protector -fstack-protector-all -fPIE -pthread -static-libstdc++ -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -o zcash/GenerateParams zcash/GenerateParams.o -fopenmp  -L/home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/share/../lib /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a -lboost_filesystem-mt -lboost_program_options-mt -lboost_thread-mt -lboost_chrono-mt libzcash.a libbitcoin_util.a crypto/libbitcoin_crypto.a -lsnark -L/home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/lib /home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/lib/libgmpxx.a /home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/lib/libgmp.a -lboost_system-mt -lcrypto /home/daira/zecc/zcash/depends/x86_64-unknown-linux-gnu/lib/libsodium.a -lrustzcash -lanl -fopenmp -pthread
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a(env.o): relocation R_X86_64_32 against `gomp_global_icv' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a: error adding symbols: Bad value

[daira]

This problem appears to be similar to one we hit before launch when trying to statically compile libsnark: #175

We resolved that by adding -fPIC to the libsnark build flags; unfortunately, we can't do the same for the libgomp.a that the system provides, because we haven't compiled it.

[daira]

I can get around the above problem by disabling PIE, which causes the resulting executable to fail the security-hardening check. This is not a good solution. An alternative would be to recompile libgomp as a static library with -fPIC, but that apparently requires a full bootstrap of gcc, and I would rather have a tooth pulled (literally, since I have a broken tooth at the moment) than deal with that if there's any alternative.

[daira]

Current state of play:

daira@sappho:~/zecc/zcash$ cat omp.cpp
#include <omp.h>
#include <iostream>

int main(int argc, char **argv) {
    int n = 0;
#pragma omp parallel
    {
        n = omp_get_num_threads();
    }
    std::cout << n << std::endl;

    return 0;
}
daira@sappho:~/zecc/zcash$ g++ -static-libstdc++ -fPIC -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fopenmp -c omp.cpp -o omp.o
daira@sappho:~/zecc/zcash$ g++ -static-libstdc++ -fPIE -fstack-protector-strong -Wl,-z,relro -Wl,-z,now omp.o $(g++ --print-file-name=libgomp.a) -ldl -pthread -o omp
daira@sappho:~/zecc/zcash$ ./omp
4
daira@sappho:~/zecc/zcash$ contrib/devtools/symbol-check.py --allow-exports omp

which looks promising, but the result does not actually have PIE enabled:

daira@sappho:~/zecc/zcash$ contrib/devtools/security-check.py omp
omp: failed PIE

Turns out the -pie option is needed, but that runs into the non-PIC libgomp problem again:

daira@sappho:~/zecc/zcash$ g++ -static-libstdc++ -pie -fPIE -fstack-protector-strong -Wl,-z,relro -Wl,-z,now omp.o $(g++ --print-file-name=libgomp.a) -ldl -pthread -o omp
/usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a(parallel.o): relocation R_X86_64_32 against `gomp_global_icv' can not be used when making a shared object; recompile with -fPIC
/usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a: error adding symbols: Bad value
collect2: error: ld returned 1 exit status

But at least now I can reproduce the problem on a small example.

[daira]

(Before anyone points it out, I know the above program technically has a race condition on n and therefore has undefined behaviour.)

[daira]

Given #3652, it might be possible to remove all of the libsnark code that uses OpenMP, and get rid of the libgomp dependency.

[daira]

libgomp was removed as a dependency by #4060. We need to revisit this with just -static-libstdc++ in the g++ build flags.

[daira]

I made some progress on this in #4613 (comment) . The current state of play is that I can produce zcash-cli and zcash-tx executables, natively compiled using Clang 8 with C++17 support on Debian 10, that are statically linked with libc++ and pass the security hardening checks. These executables don't work on Ubuntu 16.04 due to a dependency on too-recent GLIBC symbols, but I think that's a shallow problem.

$ ldd src/zcash-cli
	linux-vdso.so.1 (0x00007ffffbf58000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f5376e6f000)
	libanl.so.1 => /lib/x86_64-linux-gnu/libanl.so.1 (0x00007f5376e69000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5376d24000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f5376d0a000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f5376ce9000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5376b26000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f537724a000)

[on Ubuntu 16.04]
$ ./zcash-cli
./zcash-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by ./zcash-cli)

Unfortunately the corresponding zcashd, zcashd-gtest, etc. do depend on libc++.so.1 and libc++abi.so.1 (and librt.so.1, which is unimportant):

$ ldd src/zcashd
	linux-vdso.so.1 (0x00007ffede9dd000)
	libc++.so.1 => /usr/lib/x86_64-linux-gnu/libc++.so.1 (0x00007f2df6713000)
	libc++abi.so.1 => /usr/lib/x86_64-linux-gnu/libc++abi.so.1 (0x00007f2df66dc000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f2df66bb000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f2df66b0000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2df66ab000)
	libanl.so.1 => /lib/x86_64-linux-gnu/libanl.so.1 (0x00007f2df66a5000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2df655e000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f2df6544000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2df6381000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2df7a16000)

[daira]

-lc++ was being included twice on the linker command line, once (correctly) after -Bstatic and once (incorrectly) before. Not sure exactly why that is happening for zcashd and not for zcash-cli, but the problem is no longer mysterious and looks shallow from here.

[daira]

This will be fixed by #4613.