Async Signals

[TitanNano]

This has been developed last year in #261 and consists of two somewhat independent parts:

• A Future for Signal: an implementation of the Future trait for Godots signals.
• Async runtime for Godot: a wrapper around Godots deferred code execution that acts as a async runtime for rust futures.

The SignalFuture does not depend on the async runtime and vice versa, but there is no point in having a future without a way to execute it.

For limitations see: #261 (comment)

Example

let node = Node::new_gd();

// spawn a new async task
godot_task(async move {
    // do something before waiting for a signal
    let children = node.get_children();
    
    // await a signal
    let _: () = Signal::from_object_signal(&node, "tree_entered").to_future().await;

    // do more after the signal
   children.iter_shared().for_each(|child| ... );
});

---

TODOs

• Decide if we want to keep the GuaranteedSignalFuture. Should it be the default? (We keep it as TrySignalFuture, the plain signal is a wrapper that panics in the error case.)
• Documentation
• figure out async testing.
• deal with async panics (in tests)

CC @jrb0001 because they provided very valuable feedback while refining the POC.
Closes #261

[GodotRust]

API docs are being generated and will be shortly available at: https://godot-rust.github.io/docs/gdext/pr-1043

[Bromeon]

Thanks a lot, this is very cool!

From the title I was first worried this might cause many conflicts with #1000, but it seems like it's mostly orthogonal, which is nice 🙂

I have only seen the first 1-2 files, will review more at a later point. Is there maybe an example, or should we just check tests?

[jrb0001]

I am currently testing it with my project.

• Executor from this PR and signals from my old implementation (based on async_channel) seems to work ingame.
• My old executor (based on async_task running once per frame) and signals from this PR is the next step, hopefully tomorrow.
• Both Executor and signals from this PR will come after that. I expect some issues with recursive signals but let's see.
• I am getting a weird segfault on hotreloading with a completely useless backtrace which didn't happen with my executor implementation. I need to debug this more, but I suspect it is related to having a tool node spawn a future which listens on its signals and/or a signal>drop>signal>drop something else>signal chain.

[lilizoey]

* I am getting a weird segfault on hotreloading with a completely useless backtrace which didn't happen with my executor implementation. I need to debug this more, but I suspect it is related to having a tool node spawn a future which listens on its signals and/or a signal>drop>signal>drop something else>signal chain.

i'd guess it's related to using thread_local here which we need to do some hacky stuff to support with hot-reloading enabled

[TitanNano]

i'd guess it's related to using thread_local here which we need to do some hacky stuff to support with hot-reloading enabled

Shouldn't the hot-reload hack only leak memory? 🤔

@jrb0001 does the segfault occur on every hot-reload?

[jrb0001]

i'd guess it's related to using thread_local here which we need to do some hacky stuff to support with hot-reloading enabled

Shouldn't the hot-reload hack only leak memory? 🤔

@jrb0001 does the segfault occur on every hot-reload?

I am not completely sure yet. It doesn't happen if there are no open scenes or if none of them contains a node which spawns a Future.

It also doesn't seem to happen every single time if I close all scenes and then open one with a Future before triggering the hot-reload. In this case it panics with some scenes:

ERROR: godot-rust function call failed: <Callable>::GodotWaker::wake()
    Reason: [panic]  Future no longer exists when waking it! This is a bug!
  at /home/jrb0001/.cargo/git/checkouts/gdext-3ec94bd991a90eb6/2877010/godot-core/src/builtin/async_runtime.rs:271

With another scene it segfaults in this scenario.

Simply reopening the editor (same scene gets opened automatically) and then triggering a hot-reload segfaults for both scenes.

With both executor + Future from this PR, the hot-reload issue doesn't happen at all?!? So the issue could also be in my code, let me debug it properly before you waste more time on it.

I will do some more debugging later this week (probably weekend).

---

I also finished testing the Future part of the PR and it works fine with both my old executor and your executor in my relatively simple usage.

Unfortunately all my complex usages (recursion, dropping, etc.) need a futures_lite::Stream which I can't implement on top of your GuaranteedSignalFuture without potentially missing (or duplicating?) some signals while reconnecting with a new Future instance.

The R: Debug bound on to_future()/to_guaranteed_future() was a bit annoying and doesn't seem to be used? Or did I miss something?

[TitanNano]

The R: Debug bound on to_future()/to_guaranteed_future() was a bit annoying and doesn't seem to be used? Or did I miss something?

Yeah, it's completely unnecessary now. Probably an old artifact. I removed the bound.

---

Unfortunately all my complex usages (recursion, dropping, etc.) need a futures_lite::Stream which I can't implement on top of your GuaranteedSignalFuture without potentially missing (or duplicating?) some signals while reconnecting with a new Future instance.

Can you elaborate what the issue here is?

---

I'm also curious what your use-case for the GuaranteedSignalFuture is. Currently, I'm still thinking to get rid of it again. I have never come across a future that resolves when the underlying source disappears, and I wonder if it is really that useful for most users. But maybe you can share how it's important for you.

[TitanNano]

ERROR: godot-rust function call failed: <Callable>::GodotWaker::wake()
    Reason: [panic]  Future no longer exists when waking it! This is a bug!
  at /home/jrb0001/.cargo/git/checkouts/gdext-3ec94bd991a90eb6/2877010/godot-core/src/builtin/async_runtime.rs:271

@jrb0001 Do you have an idea what could have triggered this? The only thing that I can think of is that a waker got cloned and reused after the future resolved. The panic probably doesn't make any sense, since the waker can technically be called an infinite number of times. 🤔

[TitanNano]

@Bromeon I now added a way to test async tasks. I still need to deal with panics inside a Future, though. Technically, we could unify the test execution of sync and async tasks, but I get the impression that it also would have some downsides. Keeping it separate adds a bit of duplication, but unifying it would force more complexity onto the execution of sync tasks.

[Bromeon]

I've finally had some time to look more closely at this. Thanks so much for this great PR, outstanding work as always ❤️

Technically, we could unify the test execution of sync and async tasks, but I get the impression that it also would have some downsides. Keeping it separate adds a bit of duplication, but unifying it would force more complexity onto the execution of sync tasks.

I think you made the right choice here, it seems they're different enough to be treated differently. If it becomes bothersome in the future, we could always revise that decision; but I think keeping the sync tests simple is a good approach.

[jrb0001]

I'm also curious what your use-case for the GuaranteedSignalFuture is. Currently, I'm still thinking to get rid of it again. I have never come across a future that resolves when the underlying source disappears, and I wonder if it is really that useful for most users. But maybe you can share how it's important for you.

My experience seems to be the exact opposite of yours. Usually things like sockets and channels return Err/None/panic when the other side disappears. I don't think I have ever encountered a Future that gets stuck intentionally.

With Godot this isn't only caused by intentionally disconnecting a signal, but also when a node is freed, which can happen at any time and on a large scale. I don't like the idea of having hundreds or maybe even thousands of stuck tasks after the player changed scenes a few times.

I also think we shouldn't compare it to gdscript, for two reasons:

• gdscript doesn't need to store any additional state so it doesn't have a memory leak. Your runtime "leaks" memory through the thread-local if a task gets stuck.
• Not sure how to explain this, but for me the direction behind them is different. gdscript (and rust with Callable) is "Godot should call this method when ..." (Godot is the owner / pushing) while Future is "My future should wait until ..." (Future/Runtime is the owner / pulling). The Callable approach can detect the disconnect through NOTIFICATION_PREDELETE (gdscript) or drop() (Rust Callable) while the latter completely depends on the behavior of the signal future.

Your SignalFuture is usually enough and more ergonomic than the GuaranteedSignalFuture but I would make it panic on disconnect and make the Runtime clear the task on panic. The GuaranteedSignalFuture is still helpful if you need to wait for some signal and detect when the source disappears at the same time, without combining multiple signals, relying on catch_unwind() or a custom Drop impl.

I unfortunately didn't get to do my debugging session due to sickness. I will let you know once I have some results, but that will most likely be towards the end of the week or even weekend.

[Bromeon]

Thanks a lot for the detailed insights, @jrb0001 👍

I'm trying to see it from a user perspective. A user would then have to make a choice whether the basic future is enough or the guaranteed one is needed, which may be... not a great abstraction?

How would you advise a library user to choose correctly here, without needing to know all the details? Does the choice even make sense, or should we sacrifice a bit of ergonomics for correctness?

[TitanNano]

My experience seems to be the exact opposite of yours. Usually things like sockets and channels return Err/None/panic when the other side disappears. I don't think I have ever encountered a Future that gets stuck intentionally.

I get this point, but I wouldn't say the future gets stuck intentionally. If you create a Godot Object and don't free it, then it leaks memory. That is also not intentional. From my point of view, async tasks must be stored and canceled before freeing the Object, this is simply an inherited requirement from the manually managed Node / Object. We can put this into the documentation of the TaskHandle. Maybe we also want to make the TaskHandle #[must_use]?

I also think making the SignalFuture panic if it's Callable gets dropped would be a good compromise. This would highlight that something unexpected is happening.

[Dheatly23]

I get this point, but I wouldn't say the future gets stuck intentionally. If you create a Godot Object and don't free it, then it leaks memory. That is also not intentional. From my point of view, async tasks must be stored and canceled before freeing the Object, this is simply an inherited requirement from the manually managed Node / Object. We can put this into the documentation of the TaskHandle. Maybe we also want to make the TaskHandle #[must_use]?

But isn't manually cancelling TaskHandle is too much of a chore? Consider this simple GDScript example:

extends Button

func _pressed():
    await get_tree().create_timer(1.0).timeout
    print("Pressed one second before!")

If the button got freed, the call simply drops without any cleanup code. But with your proposal we need to store all TaskHandle in the node and cancel them all on exit tree, am i right?

Small nitpick, but i disagree on naming it GuaranteedSignalFuture, it give impression that the future will resolve without errors. I suggest naming it TrySignalFuture to emphasize that the signal might never resolve (eg. the node is removed). My potential use case is for asynchronous task cleanup like sending final message or waiting/selecting on multiple signals.

[Bromeon]

From the discussion, it's stated that the "guaranteed" future is less ergonomic to use than the regular one. At the same time, it seems like the regular one needs manual cleanup (thus being less ergonomic in its own way).

To be on the same page, could someone post similar usage examples for each of them? 🙂

[coder137]

Since SignalFuture can only get stuck in the event that the event is not fired.
What if we remove the SignalFuture and only keep the TrySignalFuture and give users a way to gracefully shutdown via CancellationToken (or something similar)? In the example it seems as though TrySignalFuture returns err once the game shuts down.

The reason I say this is because, user's might accidentally use SignalFuture's in their code without completely understanding its usecase or their own logic as the game changes.
In the event that the game hangs, they would need to go through some effort in order to debug the problem. If there are multiple SignalFuture's in the code, debugging the halts/freezes becomes harder.

[TitanNano]

@coder137 SignalFuture can no longer get stuck. It will produce a panic. So users who run into these panics can choose to either switch from the TrySignalFuture or work around the cause for the panic.

In the example it seems as though TrySignalFuture returns err once the game shuts down.

No, it happens as soon as the signal object is freed. Of course, if you wait for a signal until the game shutdown, this could also happen during shutdown. But during shutdown, all pending tasks are also being canceled and dropped before the engine cleans up the scene tree, so it's unlikely to happen.

[jrb0001]

I am not completely sure yet. It doesn't happen if there are no open scenes or if none of them contains a node which spawns a Future.

It also doesn't seem to happen every single time if I close all scenes and then open one with a Future before triggering the hot-reload. In this case it panics with some scenes:

ERROR: godot-rust function call failed: <Callable>::GodotWaker::wake()
    Reason: [panic]  Future no longer exists when waking it! This is a bug!
  at /home/jrb0001/.cargo/git/checkouts/gdext-3ec94bd991a90eb6/2877010/godot-core/src/builtin/async_runtime.rs:271

With another scene it segfaults in this scenario.

Simply reopening the editor (same scene gets opened automatically) and then triggering a hot-reload segfaults for both scenes.

With both executor + Future from this PR, the hot-reload issue doesn't happen at all?!? So the issue could also be in my code, let me debug it properly before you waste more time on it.

I will do some more debugging later this week (probably weekend).

This is still the same behavior with the current commit, both the printed panic and the segfault.

(lldb) bt
* thread #1, name = 'godot.linuxbsd.', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x7213a9b64980)
  * frame #0: 0x00007213a9b64980
    frame #1: 0x000057bde516208f godot.linuxbsd.editor.dev.x86_64`CallableCustomExtension::call(this=0x000057be33a2d150, p_arguments=0x0000000000000000, p_argcount=0, r_return_value=0x00007ffdeb3d9050, r_call_error=0x00007ffdeb3d9044) const at gdextension_interface.cpp:170:12
    frame #2: 0x000057bde4ddfb88 godot.linuxbsd.editor.dev.x86_64`Callable::callp(this=0x00007213a008a410, p_arguments=0x0000000000000000, p_argcount=0, r_return_value=0x00007ffdeb3d9050, r_call_error=0x00007ffdeb3d9044) const at callable.cpp:57:15
    frame #3: 0x000057bde51860d2 godot.linuxbsd.editor.dev.x86_64`CallQueue::_call_function(this=0x000057be174450f0, p_callable=0x00007213a008a410, p_args=0x00007213a008a428, p_argcount=0, p_show_error=true) at message_queue.cpp:220:18
    frame #4: 0x000057bde51864d5 godot.linuxbsd.editor.dev.x86_64`CallQueue::flush(this=0x000057be174450f0) at message_queue.cpp:268:20
    frame #5: 0x000057bde2aa6c73 godot.linuxbsd.editor.dev.x86_64`SceneTree::physics_process(this=0x000057be19f32eb0, p_time=0.016666666666666666) at scene_tree.cpp:492:38
    frame #6: 0x000057bde034fac7 godot.linuxbsd.editor.dev.x86_64`Main::iteration() at main.cpp:4070:60
    frame #7: 0x000057bde02866da godot.linuxbsd.editor.dev.x86_64`OS_LinuxBSD::run(this=0x00007ffdeb3d92e0) at os_linuxbsd.cpp:962:22
    frame #8: 0x000057bde027e555 godot.linuxbsd.editor.dev.x86_64`main(argc=5, argv=0x00007ffdeb3d9928) at godot_linuxbsd.cpp:85:9
    frame #9: 0x00007213bc0dbe08 libc.so.6`___lldb_unnamed_symbol3261 + 120
    frame #10: 0x00007213bc0dbecc libc.so.6`__libc_start_main + 140
    frame #11: 0x000057bde027e2f5 godot.linuxbsd.editor.dev.x86_64`_start + 37

image list before the reload, sorted by address:

[ 68] DED83DF7-4521-915A-7BD6-E8990F4F802E-13B43F04 0x00007213a7db3000 /usr/lib/libasyncns.so.0 
[ 17] 7826DD70-4046-27B0-AEBE-4E42CA605707-322A0608 0x00007213a8e00000 /home/jrb0001/GodotProjects/vn-test/rust/target/x86_64-unknown-linux-gnu/debug/libgame.so 
[ 42] DE7A5D36-BC86-167E-D7DE-538A6F63CDC5-26014ACE 0x00007213a9e68000 /usr/lib/libwayland-client.so.0 
[ 41] 81E599CC-EF35-6DCD-56EE-E45E615603EA-2D605B22 0x00007213a9e77000 /usr/lib/libxcb-dri3.so.0 

So I guess there is a Callable still alive during the reload? Is there a way to figure out which one?

---

Another issue I am currently running into is that godot_task() polls the future immediately. Is this intentional? All other wake-ups go through a call_deferred(), only the first poll is different. It also didn't happen in an older iteration of this PR.

[TitanNano]

So I guess there is a Callable still alive during the reload? Is there a way to figure out which one?

Gut feeling for the callable is that Godot is not reloading the custom callable and tries to access the invalid pointer after unloading the old library.

I can look into this more.

---

godot_task() polls the future immediately

Yes, this is intentional at the moment. There was the issue that signal futures inside an async block or function would only get created after the deferred initial poll from the godot_task and any signal that emitted right after calling godot_task would be ignored.

What exact issue is it causing?

[jrb0001]

I am calling godot_task() from IButton::pressed() and the future ends up calling Gd<MyButton>::set_focusmode() which triggers a notification and it can't create a second mut borrow to call IButton::on_notification. The first await comes somewhere after that. I need to defer somewhere, but I can't defer the call to godot_task() itself because I need to store the TaskHandle. So I guess awaiting a future which simply wakes itself is the only solution.

With my old executor, I also abused async blocks without any awaits as a generic "call deferred" solution. Simply because it was easier than messing around with Callable and its Send bound. We can ignore this usecase in my opinion.

[TitanNano]

I am calling godot_task() from IButton::pressed() and the future ends up calling Gd::set_focusmode() which triggers a notification and it can't create a second mut borrow to call IButton::on_notification. The first await comes somewhere after that. I need to defer somewhere, but I can't defer the call to godot_task() itself because I need to store the TaskHandle. So I guess awaiting a future which simply wakes itself is the only solution.

Can't you do something like this?

let base = self.base_mut();
let handle = godot_task(...); 
drop(base);
self.task_handles.push(handle);

---

Unfortunately, I wasn't able to replicate the hot-reload issue on macOS yet. It might be Linux-specific.

EDIT: can't replicate it in CI either.

[jrb0001]

Can't you do something like this?

let base = self.base_mut();
let handle = godot_task(...); 
drop(base);
self.task_handles.push(handle);

Yes, that worked. Thanks for the idea!

Unfortunately, I wasn't able to replicate the hot-reload issue on macOS yet. It might be Linux-specific.

EDIT: can't replicate it in CI either.

I think it is more likely something specific to my setup. I can't reproduce it with all my scenes, some just print panics without segfaulting. But for the affected scenes, it is 100% reproducible. I will try it in a clean project.

[jrb0001]

Minimalistic repro project: executor-segfault.zip

cd rust
cargo build --target x86_64-unknown-linux-gnu
$GODOT4_BIN --editor --path ../project/ &
# Wait until project is fully loaded.
touch game/src/lib.rs
cargo build --target x86_64-unknown-linux-gnu
# Focus editor window --> segfault.

If you are not on linux, then just change the x86_64-unknown-linux-gnu to your target. The .gdextension file is configured for anything supported by the official export templates.

So it looks like this is caused by calling the Waker inside drop() of a (tool) Node.

[TitanNano]

@jrb0001 it looks like your issue comes down to calling call_deferred on a callable during GDExtension deinitialization (The rust struct of the Node is dropped during deinitialization of the extension when hot-reloading it). Replicating your setup causes a panic for me on macOS:

unsafe precondition(s) violated: hint::unreachable_unchecked must never be reached
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread caused non-unwinding panic. aborting.

But on linux in CI it works perfectly fine (perhaps the timing is slightly different?). I think you can create a new issue for this edge-case of deferred calling callables during deinitialization.

[Bromeon]

Thanks again for the continuous effort. Another review iteration 🙂

Some organizational questions:

1. This PR adds a ton of functionality (1.2 kLOC and counting). I think at this point it makes sense to have a dedicated module instead of spilling all into godot::builtin.
   What about godot::task, similar to the previous gdnative::tasks? We can always move it around if we find it fits better into another module, but that's much easier once it is its own module already.

2. godot_task should probably be called something that contains "spawn", just like with all other async runtimes and std::thread. Maybe even just spawn. Another reason for the module, it could then be qualified as godot::task::spawn(...).

3. TrySignalFuture -- would it make sense to name it FallibleSignalFuture or so? A bit longer, but the type isn't named very often, and this seems to be the not-so-common case if I understood correctly.

[TitanNano]

1. This PR adds a ton of functionality (1.2 kLOC and counting). I think at this point it makes sense to have a dedicated module instead of spilling all into `godot::builtin`.
   What about `godot::task`, similar to the previous [`gdnative::tasks`](https://docs.rs/gdnative/latest/gdnative/tasks/index.html)? We can always move it around if we find it fits better into another module, but that's much easier once it is its own module already.

Yes, I think this is a great idea. builtin didn't feel like a great fit, but I didn't know where else to place it.

2. `godot_task` should probably be called something that contains "spawn", just like with all other async runtimes and `std::thread`. Maybe even just `spawn`. Another reason for the module, it could then be qualified as `godot::task::spawn(...)`.

I like this in combination with 1!

3. `TrySignalFuture` -- would it make sense to name it `FallibleSignalFuture` or so? A bit longer, but the type isn't named very often, and this seems to be the not-so-common case if I understood correctly.

I was thinking about FallibleSignalFuture as well, but it seemed a bit long. I'm fine with this name too.

[Bromeon]

I was thinking about FallibleSignalFuture as well, but it seemed a bit long. I'm fine with this name too.

Most of the time, people wouldn't name it directly, right? It may still appear as IDE hint on declarations, but even then, people may often directly .await on it...

I fixed the CI issues btw, the Godot-stable artifacts expired.

[Bromeon]

Thanks a lot! I also just checked the docs, the task module looks super clean!

Added some more comments, but really just small things this time.

This starts to look very good -- if no other concerns are brought up, I'd suggest merging it on the weekend 🚀