epel-7-aarch64 build failure

[jiridanek]

https://copr.fedorainfracloud.org/coprs/vbatts/bazel/build/1424821/

I couldn't find a good error message in the aarch64 build logs, the best I found was the following. Sadly, it is missing the stderr of the failed command (afaik), which is what's needed to diagnose.

+ env ./compile.sh
Building Bazel from scratcherror: Bad exit status from /var/tmp/rpm-tmp.pj9GrC (%build)
    Bad exit status from /var/tmp/rpm-tmp.pj9GrC (%build)
RPM build errors:
Child return code was: 1
EXCEPTION: [Error()]
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/mockbuild/trace_decorator.py", line 93, in trace
    result = func(*args, **kw)
  File "/usr/lib/python3.7/site-packages/mockbuild/util.py", line 776, in do_with_status
    raise exception.Error("Command failed: \n # %s\n%s" % (command, output), child.returncode)
mockbuild.exception.Error: Command failed: 
 # /usr/bin/systemd-nspawn -q -M 9fcae9c4cf984864ad38b8f5ba3e7835 -D /var/lib/mock/epel-7-aarch64-1591282967.798725/root -a -u mockbuild --capability=cap_ipc_lock --rlimit=RLIMIT_NOFILE=10240 --capability=cap_ipc_lock --bind=/tmp/mock-resolv.jf0d9l45:/etc/resolv.conf --bind=/dev/loop-control --bind=/dev/loop0 --bind=/dev/loop1 --bind=/dev/loop2 --bind=/dev/loop3 --bind=/dev/loop4 --bind=/dev/loop5 --bind=/dev/loop6 --bind=/dev/loop7 --bind=/dev/loop8 --bind=/dev/loop9 --bind=/dev/loop10 --bind=/dev/loop11 --console=pipe --setenv=TERM=vt100 --setenv=SHELL=/bin/bash --setenv=HOME=/builddir --setenv=HOSTNAME=mock --setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin --setenv=PROMPT_COMMAND=printf "\033]0;<mock-chroot>\007" --setenv=PS1=<mock-chroot> \s-\v\$  --setenv=LANG=C.UTF-8 bash --login -c /usr/bin/rpmbuild -bb --target aarch64 --nodeps /builddir/build/SPECS/bazel.spec

(from https://download.copr.fedorainfracloud.org/results/vbatts/bazel/epel-7-aarch64/01424821-bazel3/builder-live.log.gz)

NB. in the logs, there are various notices about killing orphans here and there. Good old blood-thirsty Unix.

[jiridanek]

It should be possible to reproduce this in docker (aarch64 centos 7 may be hard to get). I'm going to try the setup described in (completely unrelated, it's just where I first learned this is possible) bazelbuild/bazel#11379.

[jiridanek]

It seems to be building just fine with rpmbuild --rebuild, that is, without mock and without nspawn. My docker container didn't have systemd running... I'll try with mock and nspawn next.

[jiridanek]

Two updates. At this point in time, aarch64 epel for rhel 7 is deprecated, and qemu interpreter does not work (for me) with mock, it will probably take a full VM.

https://fedoraproject.org/wiki/EPEL
EPEL 7: x86_64, ppc64le, sources (EPEL-7 for aarch64 is no longer supported as Red Hat ended support for this architecture).

That could be good excuse to just drop aarch64 bazel on rhel7 as well... Alternatively, users can be told to run rpmbuild --rebuild .src.rpm, which works (at least for me).

multiarch/qemu-user-static#17
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --credential yes

# need newer mock and mock-core-configs from epel
yum install http://mirror.karneval.cz/pub/linux/fedora/epel/7/aarch64/Packages/e/epel-release-7-12.noarch.rpm
yum install mock rpm-build

# build with rpmbuild
curl -O https://download.copr.fedorainfracloud.org/results/vbatts/bazel/epel-7-aarch64/01424821-bazel3/bazel3-3.2.0-1.el7.src.rpm
yum-builddep bazel3-3.2.0-1.el7.src.rpm
rpmbuild --rebuild bazel3-3.2.0-1.el7.src.rpm

Build successful! Binary is here: /root/rpmbuild/BUILD/bazel-3.2.0/output/bazel
+ env ./output/bazel build --host_force_python=PY2 --nokeep_state_after_build --notrack_incremental_state --nokeep_state_after_build --sandbox_debug --host_javabase=@local_jdk//:jdk --verbose_failures //scripts:bazel-complete.bash
Opening zip "/proc/self/exe": lseek(): Bad file descriptor
FATAL: Failed to open '/proc/self/exe' as a zip file: (error: 9): Bad file descriptor
error: Bad exit status from /var/tmp/rpm-tmp.L6sINu (%build)

# build with mock
vi /etc/mock/site-defaults.cfg

useradd jacqueline
usermod -a -G mock jacqueline
su jacqueline
cd ~
curl -O https://download.copr.fedorainfracloud.org/results/vbatts/bazel/epel-7-aarch64/01424821-bazel3/bazel3-3.2.0-1.el7.src.rpm
/usr/bin/mock --root /etc/mock/epel-7-aarch64.cfg --rebuild bazel3-3.2.0-1.el7.src.rpm

qemu:handle_cpu_signal received signal outside vCPU context @ pc=0x401839
ERROR: Error in configuration

[vbatts]

please reopen is this is still an issue

[jiridanek]

I think the build would still fail on epel-7-aarch64, but at this point I am using only RHEL 9 and RHEL 8, so I don't care, and most likely nobody else does either.