Analyzing function declarations

If you want to detour/redirect a function, you need to know it's declaration (calling convention, parameters, return type). This small guide shows you how to do so by analyzing the function disassembly, in programs like IDA or Ghidra. This is made for Visual C++ x86, but might be extendable to other compilers.

Is it a thiscall?

The first thing you must identify: is the function a thiscall (that is, a class member method) or is it static?

Member methods use the ECX register to pass the this pointer to the method. So if you see the ECX register is being used before being assigned any value, there you have it: it's a thiscall.

Sometimes it's not that easy: after all, a member method does not necessarily need to use the this pointer or any of its members. Another way of knowing it is by looking at the XREFS, that is, the parts of the code that call our function. If you see they assign the ECX register before calling, then our function is a thiscall.

If the function is not a thiscall, then it's a static method.

Identifying parameters

The next important thing you need to know is the parameters your method takes. The only thing you really need to know is how many parameters the function has (or more precisely, how many bytes they use). There are two ways, depending on the type of function:

Member methods

The thiscall convention uses callee clean-up. That is, at the end of the function, in the return statement, the function is responsible of cleaning up the parameters, restoring the stack.

To know the number of parameters:

1. Find the return statement (any of them, if there is more than one)
2. Look at the number and divide by 4. If there is no number, there are no parameters.

There are multiple ways to guess the type of the parameters, described here.

Static methods

Static methods use a caller clean-up convention. That is, the code that calls the function is responsible of cleaning up the parameters and restoring the stack.

To know the number of parameters:

1. Find code that calls the function.
2. After the function call, find an instruction like add sp, NUMBER. If there's no such instruction near, there are no parameters.
3. Divide that number by four.

Warning: Sometimes the compiler merges multiple add sp... instructions in one. It's better to check more than one place.

Return value

Even though knowing the return type (if it even returns something) of the function is not strictly necessary. If you cannot guess it, our recommendation is that (at least for detouring) you use a generic int return type, just in case.

Basically, Visual C++ uses the eax register to return values. So if you see that the function assigns a value to eax, and doesn't use the register before returning the function, then that's the return value.

There are multiple ways to guess the return type, described here.