For some reason or other, long since forgotten, I once needed a test bed for arbitrarily poking into WIN32 dynamic link libraries (DLLs): that is, I wanted to be able to open a library and generically call any of its exported functions, more-or-less regardless of its arguments, return type, and calling convention. I don’t remember the exact reasons now – it’s possible I was trying to hack into some undocumented Microsoft API, but it’s also possible that I was just looking for a nice, extravagant way to say “Hello world”…
The truth was probably somewhere in between.
At least initially, I intended to support only primitive data types and stack-based calling conventions, and calling a single function at a time. Aside from the core issues around actually making the call, the needs were pretty simple. I started out with an MFC dialogue-based application. Here we have the main dialogue, having already entered all the parameters needed to call our MessageBox function (actually, in this case the ANSI version, MessageBoxA):
First, a couple of basic pieces of information: the DLL filename and the function name within the DLL, with a browse button for each.
Since I was focusing on the WIN32 API, I was really only worried about the STDCALL and C calling conventions, basically who is going to restore the stack at the end of the function call, callee or caller respectively. As we’ll see later, this choice is really more of a hint than anything. If we want to have any hope of coming out of this thing alive, we will want to check the stack afterwards anyway.
For the function’s return type as well as each of its arguments, we have a child dialogue for the details. The Argument dialogue has an extra field for the value. Only basic value types, plus C strings were supported at this point (the class and other choices were placeholders for the future and were treated the same as strings). Certain invalid combinations were prohibited, such as strings passed by value and unsigned floats.
The backing Argument class (abbreviated below) has a single union member for all the supported value types, plus an additional member for strings, and various flags for the indirection level, signed/unsigned indicator, etc:
class Argument
{
char m_nBaseType;
char m_nIndirection;
bool m_bSigned;
bool m_bReturnValue;
union {
DWORD m_dwData;
WORD m_wData;
BYTE m_cData;
TBYTE m_tData;
float m_fData;
double m_dData;
};
CString m_csData;
void *m_lpData;
void *m_lplpData;
};
Once everything is set up, we can click the [Test] button, and in this case, we get our nice “Hello World” message box:
Of course, this isn’t a terribly safe thing to do in the first place, although in practical terms the worst thing that would happen is that the whole thing would crash and then you’d have to start over and try to figure out what you did wrong. The main thing that I wanted to protect against, and diagnose, was stack cleanup.
Calling the DLL Function
First we have to load the DLL and get the address of the function we want, standard LoadLibrary and GetProcAddress stuff. The only fancy thing here is that if the function isn’t found on the first attempt, we try again with the letter ‘A’ appended, in case for example, the user has manually entered MessageBox and we really want the ANSI version. (Eventually, I would likely have made this a little smarter, to check whether it was already part of the name, and to handle the wide versions, too.)
if (!(m_hTheDll = LoadLibrary(csDllName))) {
AfxMessageBox(IDS_ERR_CANTLOADDLL, 48);
GetDlgItem(IDC_TXT_DLLNAME)->SetFocus();
return;
}
if (!(m_lpfnTheFunction = GetProcAddress(m_hTheDll, csFunctionName))) {
csFunctionName += "A";
if (!(m_lpfnTheFunction = GetProcAddress(m_hTheDll, csFunctionName))) {
AfxMessageBox(IDS_ERR_CANTGETADDRESS, 48);
GetDlgItem(IDC_TXT_FUNCTIONNAME)->SetFocus();
FreeLibrary(m_hTheDll);
return;
}
}
Before the function is called, each argument is parsed, and if a CString, the buffer is locked for direct access. In case the DLL function opens another window (as in MessageBox), we disable our own window temporarily to avoid any potential actions from the user stacking up. The function itself is called through a wrapper class method. Afterwards, we free the library and update any arguments that were passed by reference.
for (lnIndex = 0; lnIndex < m_nArgumentCount; lnIndex++) {
m_lpArguments[lnIndex]->TextToDataLock();
}
EnableWindow(FALSE);
CallDLL();
EnableWindow(TRUE);
SetFocus();
FreeLibrary(m_hTheDll);
for (lnIndex = 0; lnIndex < m_nArgumentCount; lnIndex++) {
m_lpArguments[lnIndex]->TextToDataUnlock();
}
CallDLL()
Here’s the fun part. This member function is marked naked to prevent the compiler from creating a stack frame for us. First thing we’ll do is save our this pointer and the stack pointer, before we push any arguments to the function.
bool __declspec(naked)
CDllTestDlg::CallDLL() {
__asm {
PUSHAD ;save all regs
MOV dwSaveECX, ECX ;save ECX (this pointer)
MOV dwESPBeforeArgs, ESP ;save ESP
The two calling conventions we support (C and STDCALL) are both right-to-left, so here we move to the end of the list to loop backwards. (If Pascal convention is needed then the order of the arguments can be reversed in the UI.) This code cheats a little, knowing that pointers will always be four bytes wide in this environment.
MOV EBX, [ECX.m_nArgumentCount] ;EBX <- number of arguments
LEA ESI, [ECX.m_lpArguments] ;ESI <- offset@ argument pointers
MOV EAX, EBX ;EAX <- number of arguments
SHL EAX, 2 ;EAX <- number of args * 4
MOV ECX, -4 ;ECX <- size of argument pointer
ADD ESI, EAX ;ESI <- offset@ last argument + 4
SUB ESI, 4 ;ESI <- skip back to last arg
This is the main argument pushing loop. Again, in the Win32 environment, most function arguments are a single register, or four bytes wide, including all smaller types as well as pointers. The default case of loading a single value into EAX and pushing it is handled at the end of the loop; all of the odd cases are handled first and one way or the other fall through to the last PUSH at the end. If the argument is a pointer or double pointer, then just load EAX with the appropriate one and jump to the end of the loop to push it. If the argument is a FLOAT10, then the highest WORD is zero-filled; followed by another DWORD if the type is a FLOAT10 or a FLOAT8. Then, finally, our default case from above, the lowest DWORD is pushed for all value types.
PushNextArg: ;while EBX >= 0
DEC EBX ; decrement argument count
JS NoMoreArgs
MOV EDI, [ESI] ; EDI <- pointer to argument structure
MOV DL, [EDI.m_nIndirection] ; DL <- indirection level
MOV DH, [EDI.m_nBaseType] ; DH <- base type
CMP DL, INDIR_VALUE
JZ Pass_Value
CMP DL, INDIR_DBLPOINTER
JZ Pass_DblPointer
MOV EAX, [EDI.m_lpData] ; EAX <- *argument data
JMP PushArg
Pass_DblPointer:
MOV EAX, [EDI.m_lplpData] ; EAX <- **argument data
JMP PushArg
Pass_Value_Float10:
MOVZX EAX, WORD PTR [EDI.m_dwData + 8]
PUSH EAX
Pass_Value_Float8:
MOV EAX, DWORD PTR [EDI.m_dwData + 4]
PUSH EAX
MOV EAX, DWORD PTR [EDI.m_dwData]
JMP PushArg
Pass_Value:
CMP DH, TYPE_FLOAT10
JZ Pass_Value_Float10
CMP DH, TYPE_FLOAT8
JZ Pass_Value_Float8
MOV EAX, [EDI.m_dwData] ; EAX <- argument data
PushArg:
PUSH EAX ; [push it]
ADD ESI, ECX ; ESI <- offset@ next argument
JMP PushNextArg ;loop
We save the stack pointer again after all the arguments have been pushed, get our this pointer back, get the function address and then call it. Immediately afterwards (assuming we indeed made it out alive), save the resulting stack pointer after the call, and then restore it to its value before the arguments (whether or not it was already restored by the called function).
NoMoreArgs:
MOV dwESPAfterArgs, ESP
MOV ECX, dwSaveECX
MOV EBX, [ECX.m_lpfnTheFunction]
CALL EBX
MOV dwESPAfterCall, ESP
MOV ECX, dwSaveECX
MOV ESP, dwESPBeforeArgs
Finally, we process the return value from the function. Floating point values are handled specially depending on the target width; the return value is always a FLOAT10 and may be converted down to FLOAT8 or FLOAT4. All other value types are returned in EAX, or possibly EAX:EDX for 64-bit values, so we simply save both of these into our data union.
MOV BL, [ECX.m_argReturnValue.m_nBaseType]
CMP BL, TYPE_FLOAT10
JZ Return_Float10
CMP BL, TYPE_FLOAT8
JZ Return_Float8
CMP BL, TYPE_FLOAT4
JZ Return_Float4
MOV [ECX.m_argReturnValue.m_dwData + 0], EAX
MOV [ECX.m_argReturnValue.m_dwData + 4], EDX
JMP EndMe
Return_Float10:
FSTP TBYTE PTR [ECX.m_argReturnValue.m_tData]
JMP EndMe
Return_Float8:
FSTP QWORD PTR [ECX.m_argReturnValue.m_dData]
JMP EndMe
Return_Float4:
FSTP DWORD PTR [ECX.m_argReturnValue.m_fData]
EndMe:
POPAD
PUSH 1
POP EAX
RET
}
}
Now, back from the naked wrapper function above, we check whether the stack pointer after the call matched its value either before or after the arguments were pushed, that is whether the function uses the STDCALL or C calling convention respectively. If it doesn’t match what the user selected, then we display a message and correct it. (The last case shouldn’t ever happen, where the stack pointer doesn’t match either expected position.)
if (dwESPAfterCall == dwESPBeforeArgs) {
if (m_optCallTypeValue != CALL_STDCALL) {
m_optCallTypeValue = CALL_STDCALL;
AfxMessageBox(IDS_ERR_STDCALL_CONVENTION, 48);
}
}
else if (dwESPAfterCall == dwESPAfterArgs) {
if (m_optCallTypeValue != CALL_C) {
m_optCallTypeValue = CALL_C;
AfxMessageBox(IDS_ERR_C_CONVENTION, 48);
}
}
else {
m_optCallTypeValue = CALL_UNKNOWN;
AfxMessageBox(IDS_ERR_STACK_ERROR, 48);
}