Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Godot crash on Linux when compiled with GCC 12 #100768

Closed
xsellier opened this issue Dec 23, 2024 · 14 comments · Fixed by #101004
Closed

Godot crash on Linux when compiled with GCC 12 #100768

xsellier opened this issue Dec 23, 2024 · 14 comments · Fixed by #101004

Comments

@xsellier
Copy link
Contributor

xsellier commented Dec 23, 2024

Tested versions

Launch Godot.
I compiled Godot master from git using this settings:

       _,met$$$$$gg.          binogure@binogure 
    ,g$$$$$$$$$$$$$$$P.       ----------------- 
  ,g$$P"     """Y$$.".        OS: Debian GNU/Linux trixie/sid x86_64 
 ,$$P'              `$$$.     Host: A520 AORUS ELITE 
',$$P       ,ggs.     `$$b:   Kernel: 6.11-amd64 
`d$$'     ,$P"'   .    $$$    Uptime: 8 hours, 35 mins 
 $$P      d$'     ,    $$P    Packages: 4712 (dpkg) 
 $$:      $$.   -    ,d$$'    Shell: bash 5.2.37 
 $$;      Y$b._   _,d$P'      Resolution: 2560x1440, 2560x1440 
 Y$$.    `.`"Y$$$$P"'         DE: Cinnamon 6.2.9 
 `$$b      "-.__              WM: Mutter (Muffin) 
  `Y$$                        WM Theme: cinnamon (Default) 
   `Y$$.                      Theme: Adwaita [GTK2/3] 
     `$$b.                    Icons: gnome [GTK2/3] 
       `Y$$b.                 Terminal: terminator 
          `"Y$b._             CPU: AMD Ryzen 7 5800X (16) @ 4.851GHz 
              `"""            GPU: NVIDIA GeForce RTX 3070 
                              Memory: 18014MiB / 64226MiB 
gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Using git bisect, it seems that this commit causes the crash:
be86ce3

System information

Linux binogure 6.11-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.11-1~exp1 (2024-09-19) x86_64 GNU/Linux

Issue description

Launch Godot and it crashes on a SIGSEGV:

handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.4.dev.custom_build (be86ce3103d92e4f7a625cbf11866266473ac99c)
Dumping the backtrace. Please include this when reporting the bug to the project developer.

Steps to reproduce

Launch Godot on Linux

crash-godot.mp4

Minimal reproduction project (MRP)

N/A

@xsellier xsellier changed the title [4.4-dev7] Godot crash on Linux [master] Godot crash on Linux Dec 23, 2024
@akien-mga
Copy link
Member

Using git bisect, it seems that this commit causes the crash:
be86ce3

That sounds surprising, this commit is just removing some unused headers. If the code compiles, it should be equivalent to what was before.

@akien-mga akien-mga added this to the 4.4 milestone Dec 23, 2024
@akien-mga akien-mga moved this from Unassessed to Release Blocker in 4.x Release Blockers Dec 23, 2024
@hpvb
Copy link
Member

hpvb commented Dec 26, 2024

Would you mind trying to build the binary on your computer using our own linux toolchain?

https://github.com/godotengine/buildroot

You can just get a pre-build version from the "obtaining a toolchain" part of the document.

This will tell us if this is a Debian problem, or a Debian GCC problem.

Thanks!

@xsellier
Copy link
Contributor Author

xsellier commented Dec 26, 2024

Just sharing some more info:

ldd --version
ldd (Debian GLIBC 2.36-9+deb12u7) 2.36
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

I pulled the latest master, and compiled it. Still crashing on Linux, just opening Godot crashes.

Since I've started compiling the master branch and reverted this commit: be86ce3
I'm gonna wait for it to end before trying anything else. If it works, then I'll tell you, if it does not, I'm gonna try with gcc 13.

@xsellier
Copy link
Contributor Author

Reverting be86ce3 fix the crash on Linux.
Since it's mostly headers that's been removed/moved/replaced I think it may be another underlying problem.

@xsellier
Copy link
Contributor Author

Would you mind trying to build the binary on your computer using our own linux toolchain?

https://github.com/godotengine/buildroot

You can just get a pre-build version from the "obtaining a toolchain" part of the document.

This will tell us if this is a Debian problem, or a Debian GCC problem.

Thanks!

Using this toolchain solve the issue too. So I guess I can't use gcc older than v13. It might be useful to update the documentation about building Godot for Linux and to make it matches with the toolchain tools versions:
https://docs.godotengine.org/en/stable/contributing/development/compiling/compiling_for_linuxbsd.html

@akien-mga
Copy link
Member

I suspect it might be an incremental build issue, where SCons might not properly trigger rebuilding all files that need to be built, and some old object files are reused and linked into a broken executable.

Does the problem fix itself if you do a clean build from scratch with Debian's GCC? You can clean everything in a Git clone with git clean -fxd, though note it will remove any .gitignored files, including potential IDE config files in that folder, etc. Alternatively you can clean with scons [options] -c.

@xsellier
Copy link
Contributor Author

xsellier commented Dec 28, 2024

I suspect it might be an incremental build issue, where SCons might not properly trigger rebuilding all files that need to be built, and some old object files are reused and linked into a broken executable.

Does the problem fix itself if you do a clean build from scratch with Debian's GCC? You can clean everything in a Git clone with git clean -fxd, though note it will remove any .gitignored files, including potential IDE config files in that folder, etc. Alternatively you can clean with scons [options] -c.

The first build was a clean build not an incremental one, still crashing.
While the others were incremental, just pulling the faulty commit fixed the issue.

To be clear, I'm using a VM dedicated to build Godot. So libs and binaries never get updated what so ever.
I've building Godot 4 since Godot 4.1, using this VM. Usually when Godot crashes, the first thing I do, is removing the scons DB, so it triggers a clean build. Also, the git repository is clean:

Sur la branche master
Fichiers non suivis:
  (utilisez "git add <fichier>..." pour inclure dans ce qui sera validé)
	build-consolidate.sh
	build-linux.sh
	build-windows.bat
	godot.gdkey
	modules/godotdiscord/
	modules/godoterrorhandler/
	modules/godotsteam/

aucune modification ajoutée à la validation mais des fichiers non suivis sont présents (utilisez "git add" pour les suivre)

I do have to say tho, there are indeed 3 modules that I use, and I haven't tried building Godot without them. Shell scripts are there because I use them to build Godot, and the godot.gdkey contains the encryption key I use to build Godot.

I do think that adding a link to the toolchain used by the github action to the documentation might be the solution to this issue. Saying, if it crashes but you're not using this toolchain to build Godot, then "we cannot help much". Please use the specific tools we use to build Godot.

@xsellier
Copy link
Contributor Author

xsellier commented Dec 30, 2024

As requested, a git clean and scons -c to produce a fresh build (sorry for the delay):
Image

Still crashing indeed and I don't think it's an incremental build issue.

Also, I added the following commands before compiling, just to be sure to produce a clean build

      find . -name "*.lib" -not -path "*/sdk/*" -print -delete
      find . -name "*.o" -not -path "*/sdk/*" -print -delete
      find . -name "*.obj" -not -path "*/sdk/*" -print -delete
      find . -name "*.ox" -not -path "*/sdk/*" -print -delete
      find . -name "*.a" -not -path "*/sdk/*" -print -delete
      find . -name "*.ax" -not -path "*/sdk/*" -print -delete
      find . -name "*.d" -not -path "*/sdk/*" -print -delete
      find . -name "*.so" -not -path "*/sdk/*" -print -delete
      find . -name "*.os" -not -path "*/sdk/*" -print -delete
      find . -name "*.Plo" -not -path "*/sdk/*" -print -delete
      find . -name "*.lo" -not -path "*/sdk/*" -print -delete
      find . -name "*.gen.*" -not -path "*/sdk/*" -print -delete

xsellier added a commit to xsellier/godot-docs that referenced this issue Dec 30, 2024

Verified

This commit was created on github.com and signed with GitHub’s verified signature.
Since the master branch crashes using GCC 12, I think updating the requirement to GCC 13 might be a good idea (see issue: godotengine/godot#100768 )

Also, since there's a publicly available toolchian that's easy to install, and doesn't mess up with the user's machine, I'd rather add this as an option.
@Faless
Copy link
Collaborator

Faless commented Dec 30, 2024

I'm also seeing crashes when godot is built on debian (gcc (Debian 12.2.0-14) 12.2.0), also tried with a clean build, same thing.

Stack trace for project settings:

Thread 1 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
0x0000555558ac55ae in Node::is_inside_tree (this=0x0) at scene/main/node.h:489
489		_FORCE_INLINE_ bool is_inside_tree() const { return data.inside_tree; }
(gdb) bt
#0  0x0000555558ac55ae in Node::is_inside_tree (this=0x0) at ./scene/main/node.h:489
#1  0x000055555b73ad0d in Window::set_title (this=0x0, p_title=...) at scene/main/window.cpp:281
#2  0x000055555a73e253 in ProjectManager::ProjectManager (this=0x5555640d1b60) at editor/project_manager.cpp:1142
#3  0x0000555558867679 in Main::start () at main/main.cpp:4290
#4  0x000055555878d7b0 in main (argc=1, argv=0x7fffffffdd68) at platform/linuxbsd/godot_linuxbsd.cpp:84

Stack trace with -e --path path/to/project:

0x0000555558799c22 in std::__atomic_base<unsigned long>::fetch_sub (__m=std::memory_order_acq_rel, __i=1, this=0x3f7ffffffffffff0) at /usr/include/c++/12/bits/atomic_base.h:628
628	      { return __atomic_fetch_sub(&_M_i, __i, int(__m)); }
(gdb) bt
#0  0x0000555558799c22 in std::__atomic_base<unsigned long>::fetch_sub (__m=std::memory_order_acq_rel, __i=1, this=0x3f7ffffffffffff0) at /usr/include/c++/12/bits/atomic_base.h:628
#1  SafeNumeric<unsigned long>::decrement (this=0x3f7ffffffffffff0) at ./core/templates/safe_refcount.h:87
#2  CowData<char>::_unref (this=0x5555637a8e18) at ./core/templates/cowdata.h:273
#3  0x00005555587ced4d in CowData<char>::operator= (this=0x5555637a8e18, p_from=...) at ./core/templates/cowdata.h:180
#4  0x00005555587c91aa in CharString::operator= (this=0x5555637a8e18, p_str=...) at ./core/string/ustring.h:244
#5  0x000055555cfbc693 in ShaderRD::initialize (this=0x5555637a8e10, p_variant_defines=..., p_general_defines=..., r_immutable_samplers=...) at servers/rendering/renderer_rd/shader_rd.cpp:774
#6  0x000055555cffe73f in RendererRD::GI::init (this=0x5555637a84b8, p_sky=0x5555637a7b80) at servers/rendering/renderer_rd/environment/gi.cpp:3443
#7  0x000055555cfb37a4 in RendererSceneRenderRD::init (this=0x5555637a7970) at servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:1471
#8  0x000055555cfa7ca7 in RendererCompositorRD::RendererCompositorRD (this=0x555561fae800) at servers/rendering/renderer_rd/renderer_compositor_rd.cpp:348
#9  0x00005555587cea3d in RendererCompositorRD::_create_current () at ./servers/rendering/renderer_rd/renderer_compositor_rd.h:142
#10 0x000055555cda9215 in RendererCompositor::create () at servers/rendering/renderer_compositor.cpp:42
#11 0x000055555ceba9eb in RenderingServerDefault::_init (this=0x555561825140) at servers/rendering/rendering_server_default.cpp:216
#12 0x000055555cebae46 in RenderingServerDefault::init (this=0x555561825140) at servers/rendering/rendering_server_default.cpp:255
#13 0x000055555885e1f9 in Main::setup2 (p_show_boot_logo=true) at main/main.cpp:3156
#14 0x000055555885b81e in Main::setup (execpath=0x7fffffffe0d5 "/media/SSD/Code/git/godot-dev/bin/godot.linuxbsd.editor.dev.x86_64", argc=3, argv=0x7fffffffdd40, p_second_phase=true) at main/main.cpp:2685
#15 0x000055555878d77c in main (argc=4, argv=0x7fffffffdd38) at platform/linuxbsd/godot_linuxbsd.cpp:74

Using llvm (Debian clang version 14.0.6) works fine, so indeed, it seems we broke builds with gcc 12.

@akien-mga akien-mga changed the title [master] Godot crash on Linux Godot crash on Linux when compiled with GCC 12 Dec 30, 2024
@akien-mga
Copy link
Member

The stacktrace sounds relevant for you @RandomShaper @hpvb

Still very puzzled at how a simple cleanup PR removing unused headers would cause this, but if it's a compiler bug, who knows... Maybe it was only working up until now by pure chance

Would be good to try to narrow down further what part of be86ce3 triggers this. From reading the diff I'd guess the removal of Semaphore in ResourceLoader + its use of some forward declares may be playing nasty?

@hpvb
Copy link
Member

hpvb commented Dec 31, 2024

@xsellier can you tell us how exactly you build the engine? The exact scons command you use, and the versions for gcc, linker, etc you use?

This does seem like a toolchain bug of some kind, but I'd like to know exactly what the bug is, we might be triggering it elsewhere too.

@xsellier
Copy link
Contributor Author

xsellier commented Jan 1, 2025

@xsellier can you tell us how exactly you build the engine? The exact scons command you use, and the versions for gcc, linker, etc you use?

This does seem like a toolchain bug of some kind, but I'd like to know exactly what the bug is, we might be triggering it elsewhere too.

#!/bin/bash
# This script is intended to run on Linux.

set -eo pipefail

export BUILD_REVISION=official

export NB_CPU=16
export CXX=g++
export CC=gcc

echo ""
echo "BUILDING BINARY FOR Linux 64"
echo ""
scons -j${NB_CPU} CXX=$CXX CC=$CC platform=linuxbsd target=editor debug_symbols=yes arch=x86_64 use_static_cpp=yes
echo ""
echo "BUILD DONE FOR Linux 64"
echo ""

@akien-mga
Copy link
Member

akien-mga commented Jan 1, 2025

Tested on Fedora 41 with our older buildroot using GCC 10.2.0, and it seems to work fine.
https://github.com/godotengine/buildroot/releases/tag/godot-2020.11.x-2

So at least it's not that we broke support for all GCC < 13, the problem seems to be related to Debian's version or other related dependencies specifically. Time to set up a Debian VM.

Edit: I can reproduce it in the debian:12 Docker image (with podman), and I confirm that a local revert of be86ce3 works it around.

@akien-mga
Copy link
Member

So far I've narrowed it down to the changes in core/os/ from be86ce3. Reverting only those fixes the issue in my first tests.

I thought the culprit might be the change to core/os/thread_local.h that's actually wrong - it needs Mutex to be functional, but static analysis missed it as it's only used in macros. But only reverting that file doesn't seem to fix the issue.

I'll check further when I have time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Release Blocker
Development

Successfully merging a pull request may close this issue.

5 participants