We are given a PE file that we open in DetectItEasy and see that it is packed with UPX.

We can try and unpack it with upx –d, but it couldn’t be that easy.

upx -d AntiD.exe
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2013
UPX 3.91        Markus Oberhumer, Laszlo Molnar & John Reiser   Sep 30th 2013

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx: AntiD.exe: CantUnpackException: file is modified/hacked/protected; take care!!!

Unpacked 0 files.

It looks like we will have to unpack this ourselves before we can solve it. UPX uses pushad instructions at the beginning to push the registers on to the stack so that it can retrieve them after unpacking and jumping to the original entry point. We can script IDA’s debugger to set a hardware read breakpoint at the location of the pushed registers on the stack to get us close to the OEP.

import idc
import idaapi
import idautils


idc.LoadDebugger("win32", 0)
idc.StartDebugger("", "", "")

idc.GetDebuggerEvent(WFNE_SUSP, -1);

address = idc.GetRegValue('ESP') - 1
idc.AddBptEx(address, 1, BPT_RDWR)


After we hit our breakpoint, we can remove the breakpoint and run until the tail jump that gets us to the original entry point.

popa Instruction Followed by the Tail Jump

We can take the jump to the unpacked code and then use Scylla with our new found OEP to dump the process.

We can open our unpacked executable in Binary Ninja and can see there is a path that prints the good boy message and one that prints the bad boy message. There is a function called right before the branch that checks the key and determines what path we will take.

Main Showing the Good Boy and Bad Boy paths

If we look at the function we renamed to check_key, we can see that it moves bytes on to the stack and then checks to see if the input is 16 bytes long.

The program then enters a series of anti-debugging checks that will cause the function to return 0 (FALSE) if they are triggered. Before each check, there is also a string encoding operation performed against our string.

The first anti-debugging check is a call to CheckRemoteDebuggerPresent, which checks to see if the process is being debugged.

The second anti-debugging check is a call to FindWindowW checking for a Window named OLLYDBG, which is a popular debugger used by analysts.

The third anti-debugging check is a call to IsDebuggerPresent, which checks to see if the process is being debugged.

The fourth and final anti-debugging check uses the assembly instruction rdtsc twice as a timing check to see if the process is executing slowly and probably being debugged.

If we pass all the anti-debugging checks, we end up getting the final string operation, which checks the result of all the operations against an offset in the initial buffer of bytes. If they are not equal, the function returns 0 (FALSE). But if they are equal, the result is added, which is used as the xor key in the final operation.

We can copy off the initial buffer and rewrite the operations in python, so that we can obtain the key.

buffer = [0x8C, 0xF1, 0x53, 0xA3, 0x08, 0xD7, 0xDC, 0x48, 0xDB, 0x0C, 0x3A, 0xEE, 0x15, 0x22, 
0xC4, 0xE5, 0xC9, 0xA0, 0xA5, 0x0C, 0xD3, 0xDC, 0x51, 0xC7, 0x39, 0xFD, 0xD0, 0xF8, 0x3B, 
0xE8, 0xCC, 0x03, 0x06, 0x43, 0xF7, 0xDA, 0x7E, 0x65, 0xAE, 0x80 ]

answer = []
c = 0

for i in buffer:
    for x in range(31, 127):
        a = (x ^ 0x33) & 0xFF
        a = (a + 68) & 0xFF
        a = (a ^ 0x55) & 0xFF
        a = (a - 102) & 0xFF
        a = (a ^ (c & 0xFF) & 0xFF)
        if a == i:
    c += a


When we run the script we get the key.

python solve.py