Skip to content

Harden optimized hooks, Win32 ABI usage, and lifecycle - #8

Open
Jakobzs wants to merge 3 commits into
masterfrom
agent/harden-hook-safety
Open

Jakobzs wants to merge 3 commits into
masterfrom
agent/harden-hook-safety

Conversation

@Jakobzs

@Jakobzs Jakobzs commented Jul 19, 2026

Copy link
Copy Markdown
Owner

What changed

This PR fixes three correctness and safety problems found while reviewing the wrapper.

1. Make Rust-function hooks work in optimized builds

Problem: The README and integration tests called locally defined Rust functions directly. In release builds, the optimizer could constant-fold or otherwise bypass the patched entry point, so MinHook reported success while the original function still ran.

Solution:

  • keep example/test targets out of line with #[inline(never)]
  • call them through function pointers passed through std::hint::black_box
  • run the integration suite in both debug and release profiles

Why this solves it: The opaque indirect call must go through the function address that MinHook patched, instead of allowing the compiler to replace the call with a known result. The previously failing release tests now pass on both x64 and i686.

2. Enforce and test the correct Win32 calling convention

Problem: The API-hook tests used Rust-ABI fn pointers for Win32 functions. That happened to work for the zero-argument GetCurrentProcessId test, but a Win32 function with stack arguments crashes on 32-bit Windows because extern "system" uses the platform system calling convention.

Solution:

  • change Win32 detours and trampoline types to extern "system"
  • add a regression test that hooks Sleep(u32), invokes its trampoline, and runs on i686
  • document signature, ABI, unwind, module-lifetime, and trampoline-lifetime requirements on every unsafe API
  • add x64/i686 CI coverage

Why this solves it: Detours and trampolines now use the same ABI as the target API, including the 32-bit Windows stack-cleanup convention. The argument-bearing regression test passes on both architectures and would catch the previous mismatch.

3. Make initialization ownership and teardown explicit

Problem: Two independent Once values made teardown irreversible, initialization failures panic, and uninitialize() safe even though native teardown invalidates trampoline memory. The wrapper also treated MH_ERROR_ALREADY_INITIALIZED as success and could later tear down a MinHook instance owned by another component.

Solution:

  • replace the Once values with a mutex-protected lifecycle state
  • track whether initialization is wrapper-owned or externally owned
  • serialize wrapper API transitions and return initialization errors through the existing Result APIs
  • make uninitialize() unsafe and fallible
  • permit clean reinitialization after wrapper-owned teardown
  • leave externally owned MinHook state running
  • add regression tests for reinitialization and external ownership

Why this solves it: Teardown can no longer happen through a safe API without acknowledging trampoline-lifetime requirements, native failures are returned instead of panicking, and the wrapper only calls native MH_Uninitialize for initialization it owns.

API impact

MinHook::uninitialize() is now:

pub unsafe fn uninitialize() -> Result<(), MH_STATUS>

This is an intentional breaking change because callers must prove that no detour or trampoline remains in use before native executable buffers are freed.

Validation

  • cargo fmt --all -- --check
  • Clippy with -D clippy::all on x64 and i686
  • all integration tests on:
    • x64 debug
    • x64 release
    • i686 debug
    • i686 release
  • doctests in debug and release, including i686 release

@Jakobzs
Jakobzs marked this pull request as ready for review July 20, 2026 06:01

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant