Double free or corruption involving ResourceLoader.load_threaded_request

[yvie-k]

Godot version

4.0.3

System information

Linux Ubuntu 22.04

Issue description

Program terminates with: double free or corruption (fasttop)

I captured this backtrace in GDB:

Thread 22 "godot.linuxbsd." received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffb57fa640 (LWP 45613)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140736238429760) at ./nptl/pthread_kill.c:44
44      ./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140736238429760) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140736238429760) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140736238429760, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7c42476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7c287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7c896f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7ddbb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff7ca0d7c in malloc_printerr (str=str@entry=0x7ffff7dde768 "double free or corruption (fasttop)") at ./malloc/malloc.c:5664
#7  0x00007ffff7ca2a4a in _int_free (av=0x7fff90000030, p=0x7fff90006ac0, have_lock=0) at ./malloc/malloc.c:4539
#8  0x00007ffff7ca54d3 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#9  0x0000555558435a03 in memdelete<HashMapElement<StringName, Ref<StyleBox> > > (p_class=0x7fff90006ae0) at ./core/os/memory.h:112
#10 DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > >::delete_allocation (p_allocation=0x7fff90006ae0, this=0x555561b15514)
    at ./core/os/memory.h:206
#11 HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > >::clear (this=0x555561b15508) at ./core/templates/hash_map.h:265
#12 HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > >::~HashMap (this=<optimized out>, this=<optimized out>) at ./core/templates/hash_map.h:582
#13 KeyValue<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > > >::~KeyValue (this=<optimized out>, this=<optimized out>) at ./core/templates/pair.h:82
#14 HashMapElement<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > > >::~HashMapElement (this=<optimized out>, this=<optimized out>) at ./core/templates/hash_map.h:55
#15 memdelete<HashMapElement<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > > > > (p_class=0x555561b154f0) at ./core/os/memory.h:109
#16 DefaultTypedAllocator<HashMapElement<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > > > >::delete_allocation (p_allocation=0x555561b154f0, this=0x555561b56c14)
    at ./core/os/memory.h:206
#17 HashMap<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > >, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, HashMap<StringName, Ref<StyleBox>, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Ref<StyleBox> > > > > > >::clear (this=0x555561b56c08) at ./core/templates/hash_map.h:265
#18 0x00005555584aac79 in Control::_invalidate_theme_cache (this=0x555561b56480) at scene/gui/control.cpp:2353
#19 0x00005555584e1c98 in Control::_notification (this=0x555561b56480, p_notification=<optimized out>) at scene/gui/control.cpp:3063
#20 0x0000555556c3f3ee in Control::_notificationv (p_reversed=false, p_notification=2010, this=0x555561b56480) at ./scene/gui/control.h:47
#21 Container::_notificationv (p_reversed=false, p_notification=2010, this=0x555561b56480) at ./scene/gui/container.h:37
#22 ScrollContainer::_notificationv (this=0x555561b56480, p_notification=2010, p_reversed=<optimized out>) at ./scene/gui/scroll_container.h:39
#23 0x0000555559fc5a38 in Object::notification (this=0x555561b56480, p_notification=2010, p_reversed=<optimized out>) at core/object/object.cpp:790
#24 0x0000555558331f0e in Node::propagate_notification (this=0x555561b56480, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1926
--Type <RET> for more, q to quit, c to continue without paging--c
#25 0x0000555558331f2f in Node::propagate_notification (this=0x555561b7bf00, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#26 0x0000555558331f2f in Node::propagate_notification (this=0x555561b88fe0, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#27 0x0000555558331f2f in Node::propagate_notification (this=0x555560c08e10, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#28 0x0000555558331f2f in Node::propagate_notification (this=0x5555608641b0, p_notification=2010) at scene/main/node.cpp:1929
#29 0x0000555559fc5a38 in Object::notification (this=0x555560862000, p_notification=2010, p_reversed=<optimized out>) at core/object/object.cpp:790
#30 0x0000555559d4f37c in TranslationLoaderPO::load_translation (f=..., r_error=<optimized out>) at core/io/translation_loader_po.cpp:336
#31 0x0000555559d51f1a in TranslationLoaderPO::load (this=<optimized out>, p_path=..., p_original_path=..., r_error=0x55555de35cd4, p_use_sub_threads=<optimized out>, r_progress=0x55555de35cc8, p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE) at core/io/translation_loader_po.cpp:355
#32 0x0000555559d26a5b in ResourceLoader::_load (p_path=..., p_original_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x55555de35cd4, p_use_sub_threads=false, r_progress=0x55555de35cc8) at core/io/resource_loader.cpp:214
#33 0x0000555559d2718a in ResourceLoader::_thread_load_function (p_userdata=0x55555de35c98) at core/io/resource_loader.cpp:241
#34 0x0000555559bc6d66 in Thread::callback (p_caller_id=<optimized out>, p_settings=..., p_callback=0x555559d27060 <ResourceLoader::_thread_load_function(void*)>, p_userdata=0x55555de35c98) at core/os/thread.cpp:65
#35 0x000055555a8ac074 in execute_native_thread_routine ()
#36 0x00007ffff7c94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#37 0x00007ffff7d26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

I'm also getting a segfault in another thread, which dereferences a null pointer in the theme cache:

Thread 19 "godot.linuxbsd." received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb6ffd640 (LWP 49020)]
0x00005555584b0a9c in HashMap<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > >, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > > > > >::_lookup_pos (r_pos=<synthetic pointer>: 0, p_key=..., this=0x555561f6abf0) at ./core/templates/hash_map.h:119
119                             if (hashes[pos] == hash && Comparator::compare(elements[pos]->data.key, p_key)) {

(gdb) bt
#0  0x00005555584b0a9c in HashMap<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > >, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > > > > >::_lookup_pos (
    r_pos=<synthetic pointer>: 0, p_key=..., this=0x555561f6abf0) at ./core/templates/hash_map.h:119
#1  HashMap<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > >, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, HashMap<StringName, Color, HashMapHasherDefault, HashMapComparatorDefault<StringName>, DefaultTypedAllocator<HashMapElement<StringName, Color> > > > > >::has (p_key=..., 
    this=<optimized out>, this=<optimized out>, p_key=...) at ./core/templates/hash_map.h:310
#2  Control::get_theme_color (this=<optimized out>, p_name=..., p_theme_type=...) at scene/gui/control.cpp:2510
#3  0x000055555847141e in Button::_update_theme_item_cache (this=0x555561f6a3c0) at scene/gui/button.cpp:64
#4  0x00005555584e1ca5 in Control::_notification (this=0x555561f6a3c0, p_notification=<optimized out>) at scene/gui/control.cpp:3064
#5  0x00005555578daabe in Control::_notificationv (p_reversed=false, p_notification=2010, this=0x555561f6a3c0) at ./scene/gui/control.h:47
#6  BaseButton::_notificationv (p_reversed=false, p_notification=2010, this=0x555561f6a3c0) at ./scene/gui/base_button.h:40
#7  Button::_notificationv (this=0x555561f6a3c0, p_notification=2010, p_reversed=<optimized out>) at ./scene/gui/button.h:38
#8  0x0000555559fc5a38 in Object::notification (this=0x555561f6a3c0, p_notification=2010, p_reversed=<optimized out>) at core/object/object.cpp:790
#9  0x0000555558331f0e in Node::propagate_notification (this=0x555561f6a3c0, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1926
#10 0x0000555558331f2f in Node::propagate_notification (this=0x555561f66350, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#11 0x0000555558331f2f in Node::propagate_notification (this=0x555561f55400, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#12 0x0000555558331f2f in Node::propagate_notification (this=0x555561f29fc0, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#13 0x0000555558331f2f in Node::propagate_notification (this=0x555561f28990, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#14 0x0000555558331f2f in Node::propagate_notification (this=0x555561f27490, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#15 0x0000555558331f2f in Node::propagate_notification (this=0x555561f26170, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#16 0x0000555558331f2f in Node::propagate_notification (this=0x5555618c5e40, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#17 0x0000555558331f2f in Node::propagate_notification (this=0x555561aaa4c0, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#18 0x0000555558331f2f in Node::propagate_notification (this=0x555560e4ae00, p_notification=p_notification@entry=2010) at scene/main/node.cpp:1929
#19 0x0000555558331f2f in Node::propagate_notification (this=0x555560862130, p_notification=2010) at scene/main/node.cpp:1929
#20 0x0000555559fc5a38 in Object::notification (this=0x555560869770, p_notification=2010, p_reversed=<optimized out>) at core/object/object.cpp:790
#21 0x0000555559d4f37c in TranslationLoaderPO::load_translation (f=..., r_error=<optimized out>) at core/io/translation_loader_po.cpp:336
#22 0x0000555559d51f1a in TranslationLoaderPO::load (this=<optimized out>, p_path=..., p_original_path=..., r_error=0x555562efd154, p_use_sub_threads=<optimized out>, r_progress=0x555562efd148, 
    p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE) at core/io/translation_loader_po.cpp:355
#23 0x0000555559d26a5b in ResourceLoader::_load (p_path=..., p_original_path=..., p_type_hint=..., p_cache_mode=ResourceFormatLoader::CACHE_MODE_REUSE, r_error=0x555562efd154, p_use_sub_threads=false, r_progress=0x555562efd148)
    at core/io/resource_loader.cpp:214
#24 0x0000555559d2718a in ResourceLoader::_thread_load_function (p_userdata=0x555562efd118) at core/io/resource_loader.cpp:241
#25 0x0000555559bc6d66 in Thread::callback (p_caller_id=<optimized out>, p_settings=..., p_callback=0x555559d27060 <ResourceLoader::_thread_load_function(void*)>, p_userdata=0x555562efd118) at core/os/thread.cpp:65
#26 0x000055555a8ac074 in execute_native_thread_routine ()
#27 0x00007ffff7c94b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#28 0x00007ffff7d26a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

From the backtrace, it seems that when Translations are loaded, a notification is sent to the tree (perhaps to update translations). This is probably the culprit as multithreaded access to the scene is not allowed IIRC

Steps to reproduce

Reproduction steps are currently not 100% clear to me. I'm currently trying to create a MRP, but I cannot reproduce it outside of my project at the moment.

However, it seems to involve these components:

• Loading a translation in a background thread
• Freeing (queue_free) a gui node while the background thread is loading

Minimal reproduction project

WIP

[yvie-k]

Can reproduce this issue on the current master (3710f06) as well

[akien-mga]

CC @bruvzg @RandomShaper

[YuriSizov]

May be related to #63461.

[yvie-k]

I couldn't manage to produce a MRP.

However, I tested commenting out this line:

godot/core/io/translation_loader_po.cpp

Line 336 in 3710f06

translation->set_locale(value);

and both the double free and the segfault disappear.

This confirms my suspicion that this line must not be called from the loading thread:

godot/core/string/translation.cpp

Line 86 in 2930fbe

OS::get_singleton()->get_main_loop()->notification(MainLoop::NOTIFICATION_TRANSLATION_CHANGED);