[Windows] Avoid GetModuleHandle(NULL)

[AronParker]

• Tested on all platforms changed
• Added an entry to CHANGELOG.md if knowledge of this change could be valuable to users
• Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• Created or updated an example program if it would help users understand this functionality
• Updated feature matrix, if new features were added or implemented

Fixes #2300.

winit's GetModuleHandle(NULL) returns an invalid instance handle if it were to be used inside of a DLL or other non-executable target.. Using the pseudo-variable of the microsoft linker not only reduces work, but also works reliably inside both DLLs and executables. More info is inside the issue.

The documentation states that this only works with the microsoft linker, I've tested it using both the stable-GNU and stable-MSVC toolchain and it both seems to work, which suggests that on windows the microsoft linker is always used.

[filnet]

As mentioned, this works when using a Microsoft linker.
What about MSYS2/Mingw and assimilated environments ?
Will they provide a Microsoft compatible linker ?

[AronParker]

I've put together a little executable that tests this:

#include <Windows.h>
#include <stdio.h>

extern IMAGE_DOS_HEADER __ImageBase;

int main() {
	HINSTANCE imgb = (HINSTANCE)&__ImageBase;
	HINSTANCE modh = GetModuleHandle(NULL);

	printf("ImageBase: %p\n", imgb);
	printf("GetModuleHandle: %p\n", modh);

	if (imgb == modh) {
		printf("SUCCESS: pointers are equivalent");
	} else {
		fprintf(stderr, "ERROR: pointer mismatch");
	}

	return 0;
}

running it on MSYS2 gives me the following output:

Aron@DESKTOP-6CIB9GL MINGW64 ~
$ gcc test.c -o test

Aron@DESKTOP-6CIB9GL MINGW64 ~
$ ./test
ImageBase: 00007ff783760000
GetModuleHandle: 00007ff783760000
SUCCESS: pointers are equivalent
Aron@DESKTOP-6CIB9GL MINGW64 ~
$ gcc --version
gcc.exe (Rev1, Built by MSYS2 project) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Sadly my knowledge about windows linkers isn't deep enough, but my gut feeling is that everyone who went through building a windows linker directly or indirectly implements this "feature".

[filnet]

Looks good then. I was mostly concerned about MSSY2/Mingw...

Maybe we could fallback to the old behavior when required (should there be a way to detect that).
Not simple since we don't even know if such environments exist at all...

[filnet]

I went ahead and tested a rust equivalent of your test (rustc does not use gcc but llvm/clang afaik).

use windows_sys::Win32::Foundation::HINSTANCE;
use windows_sys::Win32::System::LibraryLoader::GetModuleHandleW;
use windows_sys::Win32::System::SystemServices::IMAGE_DOS_HEADER;

extern "C" {
    static __ImageBase: IMAGE_DOS_HEADER;
}

fn main() {
    let imgb = unsafe { &__ImageBase as *const IMAGE_DOS_HEADER as HINSTANCE };
    let modh = unsafe { GetModuleHandleW(std::ptr::null()) };

    println!("ImageBase: {}", imgb);
    println!("GetModuleHandle: {}", modh);

    if imgb == modh {
        println!("SUCCESS: pointers are equivalent");
    } else {
        println!("ERROR: pointer mismatch");
    }
}

Works on MINGW64 and on CLANG64 (not shown):

Philippe@DESKTOP MINGW64 ~/rust/handle
$ cargo run
   Compiling handle v0.1.0 (C:\msys64\home\Philippe\rust\handle)
    Finished dev [unoptimized + debuginfo] target(s) in 1.82s
     Running `target\debug\handle.exe`
ImageBase: 140698631536640
GetModuleHandle: 140698631536640
SUCCESS: pointers are equivalent

[AronParker]

Looks good then. I was mostly concerned about MSSY2/Mingw...

Maybe we could fallback to the old behavior when required (should there be a way to detect that). Not simple since we don't even know if such environments exist at all...

Sadly I believe implementing a fallback would be quite difficult. There are two cases in which this wouldn't work:

1. __ImageBase is not defined:

I have never seen this in practice, might be worth to get in touch with some people who know a bit more about linkers on Windows but I've never seen it fail anywhere. As far as I know, there is no way to detect whether FFI externs are defined at compile-time. The good thing is though, should this trick fail clients will be notified by compile-time errors so it would be quickly noticed and fixed.

2. __ImageBase address is wrong:

This can only happen if someone redefined the variable. This is disallowed by the standard anyway (as variable names prefixed with two underscores are reserved for the compiler/implementation). I suppose in debug mode one could print a warning if __ImageBase diverges from GetModuleHandle(NULL), although in some cases that is to be expected (e.g. when running inside a DLL).

I have a high degree of confidence that this will be functional and an improvement over the current approach, however I am not an expert on Windows linkers so that's why I put it up as a draft for now. Might be worth looking into what other major libraries use this library to increase confidence in using it.

[aloucks]

There seems to be another technique described in that stackoverflow link that doesn't depend on the linker. Has there been any experimentation with this? (I have not tried it myself)

https://stackoverflow.com/a/56804892

HMODULE getThisModuleHandle()
{
    //Returns module handle where this function is running in: EXE or DLL
    HMODULE hModule = NULL;
    ::GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS | 
        GET_MODULE_HANDLE_EX_FLAG_UNCHANGED_REFCOUNT, 
        (LPCTSTR)getThisModuleHandle, &hModule);

    return hModule;
}

[maroider]

__ImageBase has been around for a long time (check out this blog post by Raymond Chen from 2004). I'd be surprised if we encountered any issues with it.

EDIT: I didn't read the contents of the PR all that closely, so I didn't see the blog post being already linked, lol.

[AronParker]

The GetModuleHandleEx approach is clever and would probably work but there is another advantage of the __ImageBase: It does not require a syscall, it has the overhead of a global variable access which makes it the most efficient and fastest option.

Should there be a concern about it not working (even though we have yet to find a platform on where this doesn't work on - compatibility layers around windows have existed for decades and as maroider stated it's probably 2 decades old at this point so it's probably stable), we could lock it behind a feature gate. I'd still be in favour of making __ImageBase the default, as it is the more cost efficient option. Should it not be working, the GetModuleHandleEx approach could be used.

One of the big advantages is that this approach cannot silently fail. If __ImageBase is missing there'll be a compiler error, if its address is wrong, window creation will fail due to an invalid HINSTANCE. I believe it would be worth it to test this feature and should there be any issues this commit can be instantly reverted and we can get back to the drawing board and figure out a hybrid approach.

[maroider]

I think this is mostly fine. I haven't tested it myself, but I think this should be a case of "if it compiles, it works".

[msiglreith]

thanks!