We are given a 32-bit Windows console file where you have to type in the right flag to solve the challenge. After we have opened the file in our favorite disassembler, we can see the file uses a TLS callback which is executed before the main entry point:

First, the callback function checks if the process is being debugged by calling the IsDebuggerPresent() API function. If a debugger is attached to the process and there are no hiding plugins present, the function returns true. Then, it checks if the same function is being hooked by comparing the first byte of it to the following commonly used hooking instructions:

-   0xE8 – CALL

-   0xE9 / 0xEB – JMP

-   0xCC – INT 3

This is done to detect some debugger hiding plugins which hook IsDebuggerPresent() to always return false. Next, it manually checks if the “BeingDebugged” member of the Process Environment Block is set, the same what IsDebuggerPresent() does in the background. At last, it checks if a DLL named “HideDebugger.dll” is loaded into the process address space by using GetModuleHandleW() API function. This library is used by IDAStealth plugin for hiding the debugger.

If one of those checks was successful, one of 4 global variables is set either to one of the following values:

-   0x1

-   0x17

-   0x2A

The global variables are later used to silently overwrite the right flag during the encryption, but more on this later.

One way to circumvent these anti debug tricks is just to use a disassembler and not a debugger.

When we move to the main entry point we can see at the beginning a big array is filled with seemingly random data. However, these bytes are SHA-1 hashes of teach character of the flag.

After the array is filled with the hashed flag a thread is created which contains the same anti debug tricks like in the TLS callback function. But this time there is some additional code to also patch the encryption algorithm to use the right encryption key.

It does this by searching for the byte sequence 0x80C137 in the encryption function sub_46D7D0 and patch the last byte to 0x13. Without this patch, a wrong flag would be calculated and also one cannot just easily patch the execution of the thread itself.

If we follow the execution flow, we can see an event named “laby” is being created. Subsequently, the activities in function sub_46E030 are none relevant and are used to confuse the participant. There we can see some dummy calculations with random values and a couple of clear text (e.g. “lol”) and Base64 encoded strings which look like part of the flag calculation. The following encoded strings exist and give a message from the author:

SSBoYXZl (“I have”)

bm8gaWRlYQ== (“no idea”)

d2hhdCBJbSBkb2luZw== (“what Im doing”)

c3JzbHksIGxvbA== (“srsly, lol”)

The most important part is the thread which handles and encrypts the user input and calculates the SHA-1 hashes (sub_46E1F0). First, this thread creates another thread (sub_46D7D0) which contains the encryption of the user input.

We can see the anti-debug variables which got filled either with 0 (no anti debug) or the values described above (anti debug) then are used as part of the encryption. If they were filled earlier the flag will be incorrect and the participant is unable to solve the challenge even if he might have found the correct flag.

Subsequently, the SHA-1 hash of every character gets calculated which can be recognized by the 32-bit initialization constants (sub_413220, sub_4132D0). You can also use a plugin like “signsearch” (https://sourceforge.net/projects/idasignsrch) which searches for all kind of encryption algorithms in a file.

At last, every encrypted and hashed character of the user input gets compared to the ones stored in the big array at the beginning of the program.

Solution

The easiest and most quick solution is to write a small brute forcer which encrypts and hashes every printable character of the alphabet and compare it subsequently to the SHA-1 hashes inside the CTF challenge.

Therefore, we can you our favorite debugger, load the application and let it fill in the array with all the hashes. Next, we can easily dump the appropriate memory region and save the bytes to a file. The file should be 2,860 bytes in size which corresponds to a string of 143 characters (2,860 / 20 = 143).

Next, we can code a small program or script which tries all combinations of printable characters, encrypt and hash them. The final hashed character can then be compared to the ones we stored in the file. As SHA-1 hashes are 20 characters in length, we can just increment by the length of a hash and subsequently walk through the stored raw hashes file.

Example in Python:

import binascii
import hashlib

def encrypt(val):
    val = (val ^ (val >> 1)) + ord("\x13")
    
    if (val >= 128 and val <= 254):
        val = val + 1
    
    return chr(val)

with open("raw_hashes", "rb") as file:
    raw_data = bytearray(file.read())

res = ""

for i in range(0, len(raw_data)/20):
    for j in range(32, 127):
        m = hashlib.sha1()
        m.update(encrypt(j))
        
        hash = raw_data[i*20:(i+1)*20]
        
        if m.hexdigest() == binascii.hexlify(hash):
            res += chr(j)
        
print("Result:\n{}".format(res))

Console output:

So the final flag is:

PANW{The XOR operator is extremely common as a component in more complex ciphers… but you have probably brute forced it so it doesn’t matter}