c# - SetWindowsHookEx fails with error 126

Question

I'm trying to use the Gma.UserActivityMonitor library in a project and I've faced an error I can not overcome on my own.

In the HookManager.Callbacks.cs file there's a static method called EnsureSubscribedToGlobalMouseEvents with the following code (more or less):

var asm = Assembly.GetExecutingAssembly().GetModules()[0];
var mar = Marshal.GetHINSTANCE(asm);
s_MouseHookHandle = SetWindowsHookEx(
    WH_MOUSE_LL,
    s_MouseDelegate,
    mar,
    0);
//If SetWindowsHookEx fails.
if (s_MouseHookHandle == 0)
{
    //Returns the error code returned by the last unmanaged function called using platform invoke that has the DllImportAttribute.SetLastError flag set. 
    int errorCode = Marshal.GetLastWin32Error();
    //do cleanup

    //Initializes and throws a new instance of the Win32Exception class with the specified error. 
    throw new Win32Exception(errorCode);
}

The SetWindowsHookEx always returns 0 and the above code keeps throwing an exception with message The specified module could not be found and the call to Marshal.GetLastWin32Error returns code 126. I can successfully run the demo provided with Gma.UserActivityMonitor's original project but since my project is a little too complicated to explain here I can not go into detail explaining its difference with mine. I'm just hoping someone can blind guess the problem.

BTW, in the project's FAQ it's said that others have a problem close to mine (with SetWindowsHookEx returning error) when the Enable Visual Studio hosting process is checked only when the project is debugged. So I unchecked that box in mine and still I'm having the same problem, and not just in debugging mode but also when I double click the release file in Windows Explorer (no Visual Studio involved).

To give more information, in demo project (which works fine) the asm variable points to {Gma.UserActivityMonitor.dll} and the same in my project which the exception is thrown!

Answer 1

This kind of code does not work anymore on .NET 4 and up. The error code you get is otherwise descriptive, 126 = "The specified module could not be found". Which tells you that the "mar" variable contains junk.

.NET 4 had a pretty significant CLR change, it no longer pretends that jitted code lives inside unmanaged modules. So Marshal.GetHINSTANCE() does not work anymore. The code then gets sloppy, it forgets to check the return value, testing it for (IntPtr)-1 is required to detect failure and declare disaster. Pretty common for code you find at Codeproject, lots of bugs and sloppiness that can't be fixed by contributors. Not the SO model :)

SetWindowsHookEx() is a bit awkward for the low-level hooks. It requires a valid module handle, and checks it, but doesn't actually use it. This got fixed in Windows, somewhere around Win7 SP1. While certainly intended to be a useful fix, it actually made the problem worse. Because now it may work on your dev machine but not on your user's machine.

Anyhoo, the fix is simple, you just need to cough up a valid module handle. You can get one from a module that is always present in a managed app, you'll need to pinvoke LoadLibrary to get it:

var mar = LoadLibrary("user32.dll");
s_MouseHookHandle = SetWindowsHookEx(
    WH_MOUSE_LL,
    s_MouseDelegate,
    mar,
    0);

No need to call FreeLibrary(), that module stays loaded until your program terminates anyway.