Improve the design of StreamSafeCUDAAllocator

[From00]

PR types

New features

PR changes

Others

Describe

This PR do the following things to improve the StreamSafeCUDAAllocator impletemented in #37290 :

1. Rewrites the StreamSafeCUDAAllocator to directly create and record an event when the RecordStream is called, not when the allocation is freed
2. Changes the default stream of StreamSafeCUDAAllocator from NULL stream to the main stream in CUDADeviceContext
3. Adds a GetStream interface for getting the owning stream of a StreamSafeCUDAAllocation

Why we no longer create and record events in a deferred manner?
When the main process finish, Paddle may clear some remaining tensors in shared memory (see #22541), and thus free allocations that may be recorded some CUDA streams. However, the stream created in executor has already destroyed at that time, so we should not try to record events to it. Similar problems may occur in any scenarios where StreamSafeCUDAAllocator is used, so we cannot assume that the recorded streams are still valid when freeing CUDA allocations.

[paddle-bot-old]

✅ This PR's description meets the template requirements!
Please wait for other CI results.

[paddle-bot-old]

Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

[zhiqiu]

LGTM