We are given a Windows 32bit PE file that utilizes VMware backdoor I/O ports for this challenge. VMware backdoor functions are the same methods that VMware tools uses for the guest to communicate with the host. The figure below shows the backdoor function that the challenge uses where it passes the command number to the function as an argument and sets up the registers for the VMware backdoor function before calling the in instruction.

Figure 1 VMWare Backdoor Function

The first check we have to pass is if we are using VMware and the BDOOR_CMD_GETHWVERSION returns with 4. The second checks if our processor speed from BDOOR_CMD_GETMHZ is above 1000.

Figure 2 VMware Version and Processor MHZ checks

The next backdoor check is for the BIOS UUID using BDOOR_CMD_GETUUID. The result needs to equal “5f56604a5b4e29554a4b625b4e575d51”. We can patch this to pass the check.

Figure 3 BIOS UUID Check

The next backdoor commands are for BDOOR_CMD_GETTIME and BDOOR_CMD_GETSCREENSIZE. The year needs to be 2017 and the month needs to be August (08).

Figure 4 Check Year and Month

Next the BIOS UUID from earlier is used in an encoding loop with a byte array to decode the first part of the decryption key that we need for later.

Figure 5 First part of key decoding with BIOS UUID and byte array

The next section uses the backdoor check BDOOR_CMD_GETPTRLOCATION to obtain the location of the cursor. It then performs some calculations and needs the result to be 0xd and then it will give us the second part of the decryption key.

Figure 6 Mouse Cursor check

The next part uses BDOOR_CMD_GETSELLENGTH and BDOOR_CMD_GETNEXTPIECE to get the contents of the clipboard. The contents of the clipboard are compared with the length 64 and then they it is used to decode the third part of the decryption key. Later in the program the strtoul is applied to clipboard contents and 7 is added to each byte. There is a check to make sure the result equals “5168796c7b6f33277075276f707a27767e73276d76797433277e687b6a6f6c7a”. We can then work backwards and know that we need to have the hex string “4A61726574682C20696E20686973206F776C20666F726D2C2077617463686573” in the clipboard at the beginning. Interestingly, ascii characters for the hex string would be “Jareth, in his owl form, watches”.

Figure 7 Clipboard check

The fourth part of the decryption key is decoded using Fibonnaci with the month that we obtained earlier that is multiplied by 10 and then XOR’d with 0xB06B558B.

Figure 7 Fibonnaci Function

The fifth and final part of the decryption key is derived from the year we obtained earlier that has some bitwise encoding performed on it.

Figure 8 Code to decode the final part of the decryption key

The five parts of the key are then combined into the final decryption key. The decryption key is then used with rc4 to decrypt the solution. The first 4 letters of key are checked to see if they are “PAN{“ and then the flag is printed.

flag: PAN{VMWare Labyrenth 2017 Challenge. VMWare Backdoor API is nice.}

Figure 9 The parts of the decryption key are combined

Figure 10 The solution is rc4 decrypted and then printed