Typically hooks are inserted while software is already running, but hooking is a tactic that can also be employed prior to the application being started. Both these techniques are described in greater detail below.
Hooking can be achieved by modifying the source of the executable or library before an application is running, through techniques of reverse engineering. This is typically used to intercept function calls to either monitor or replace them entirely.
For example, by using a disassembler, the entry point of a function within a module can be found. It can then be altered to instead dynamically load some other library module and then have it execute desired methods within that loaded library. If applicable, another related approach by which hooking can be achieved is by altering the import table of an executable. This table can be modified to load any additional library modules as well as changing what external code is invoked when a function is called by the application.
An alternative method for achieving function hooking is by intercepting function calls through a wrapper library. A wrapper is a version of a library that an application loads, with all the same functionality of the original library that it will replace. That is, all the functions that are accessible are essentially the same between the original and the replacement. This wrapper library can be designed to call any of the functionality from the original library, or replace it with an entirely new set of logic.
Operating systems and software may provide the means to easily insert event hooks at runtime. It is available provided that the process inserting the hook is granted enough permission to do so. Microsoft Windows for example, allows users to insert hooks that can be used to process or modify system events and application events for dialogs, scrollbars, and menus as well as other items. It also allows a hook to insert, remove, process or modify keyboard and mouse events. Linux provides another example where hooks can be used in a similar manner to process network events within the kernel through NetFilter.
When such functionality is not provided, a special form of hooking employs intercepting the library function calls made by a process. Function hooking is implemented by changing the very first few code instructions of the target function to jump to an injected code. Alternatively on systems using the shared library concept, the interrupt vector table or the import descriptor table can be modified in memory. Essentially these tactics employ the same ideas as those of source modification, but instead altering instructions and structures located in the memory of a process once it is already running.
Whenever a class defines/inherits a virtual function (or method), compilers add a hidden member variable to the class which points to a virtual method table (VMT or Vtable). Most compilers place the hidden VMT pointer at the first 4 bytes of every instance of the class. A VMT is basically an array of pointers to all the virtual functions that instances of the class may call. At runtime these pointers are set to point to the right functions, because at compile time, it is not yet known if the base function is to be called or if an overridden version of the function from a derived class is to be called (thereby allowing for polymorphism). Therefore, virtual functions can be hooked by replacing the pointers to them within any VMT that they appear. The code below shows an example of a typical VMT hook in Microsoft Windows, written in C++.1
All virtual functions must be class member functions, and all (non-static) class member functions are called with the __thiscall calling convention (unless the member function takes a variable number of arguments, in which case it is called with __cdecl). The __thiscall calling convention passes a pointer to the calling class instance (commonly referred to as a "this" pointer) via the ECX register (on the x86 architecture). Therefore, in order for a hook function to properly intercept the "this" pointer that is passed and take it as an argument, it must look into the ECX register. In the above example, this is done by setting the hook function (hkVirtualFn1) to use the __fastcall calling convention, which causes the hook function to look into the ECX register for one of its arguments.
Also note that, in the above example, the hook function (hkVirtualFn1) is not a member function itself so it cannot use the __thiscall calling convention. __fastcall has to be used instead because it is the only other calling convention that looks into the ECX register for an argument.
The following example will hook into keyboard events in Microsoft Windows using the Microsoft .NET Framework.
The following source code is an example of an API/function hooking method which hooks by overwriting the first six bytes of a destination function with a JMP instruction to a new function. The code is compiled into a DLL file then loaded into the target process using any method of DLL injection. Using a backup of the original function one might then restore the first six bytes again so the call will not be interrupted. In this example the win32 API function MessageBoxW is hooked.2
This example shows how to use hooking to alter network traffic in the Linux kernel using Netfilter.
The following code demonstrates how to hook functions that are imported from another module. This can be used to hook functions in a different process from the calling process. For this the code must be compiled into a DLL file then loaded into the target process using any method of DLL injection. The advantage of this method is that it is less detectable by antivirus software and/or anti-cheat software, one might make this into an external hook that doesn't make use of any malicious calls. The Portable Executable header contains the Import Address Table (IAT), which can be manipulated as shown in the source below. The source below runs under Microsoft Windows.
psyfl https://github.com/psyfl/Virtual-Method-Table-Hooking/blob/master/ConsoleApplication5/ConsoleApplication5.cpp ↩
For more information, see http://ntvalk.blogspot.nl/2013/11/hooking-explained-detouring-library.html http://ntvalk.blogspot.nl/2013/11/hooking-explained-detouring-library.html ↩