Dissecting Windows Malware Series – Understanding Cryptography And Data Encoding

In the previous article https://8ksec.io/dissecting-windows-malware-series-process-injections-part-2/, we introduced the mechanism of Process Injection that malware use to achieve Stealth and Evasion. We saw direct implementation of:

• Process Injection
• Process Hollowing

And the use of other interesting Stealth associated mechanism.

What’s In It For Me❓

We’ll see how malware utilize complex encryption algorithms like AES and “seemingly simpler” encryption methods like Base64 to conceal their functionalities.

Attackers do this to make it harder for Malware Analysts to fully understand what the malware does and how it operates.

An interesting footnote – in many cases, attackers prefer using simpler encryption methods like Base64 Encoding or XOR Encoding because they are:

1. Space-efficient – they can be used in space-constrained environments and even in various Shellcodes.
2. Less noticeable to the eye – they do not require calling functions from the cryptographic API of a specific encryption, thus are more “quiet” in the eyes of the Analyst.
3. Efficient – because these are relatively compact encryption methods that don’t require a large number of code lines and/or calling external API functions, they less affect the performance of the malware during execution.

Following this article, you’ll be able to spot AES encryption, XOR encoding and Base64 encoding mechanisms in the sample you are analyzing, much easier.

A Brief Overview Of Encryption Methods📚

XOR Encoding

Arguably the most by-the-book method used, in one way or another, among malware.

In its simplest form, it uses a character (byte) that serves as a key, for example: 0x3C.

If we perform a XOR operation with the key on each character in our string, we will get a completely different representation of the existing string, meaningless in human eyes.

For example, the address: http[:]//badsite[.]com with the XOR operation, using the key – 0x2d on each character will translate to – VJJN…_ZMWJ[.]QS.

Overall, it doesn’t sound so terrible, right?

The issue becomes complicated when playing around with this method😈

For instance, given an ASCII string representing an IP address, a domain, or even names of functions themselves used by the malware – we could use the content of the string itself for encryption.

We could start the XOR operation at the beginning of the string, at the end or literally at any position we choose – On a randomly chosen character.

The output character we get from the XOR operation will serve us as a “key” to perform the XOR operation with the next character in the string and so on…

This method opens up a vast array of encoding possibilities for the attacker, using a simple XOR operation, starting from a specific character in the string, using nested loops from a certain position, and so on…

Base64 Encoding

Like the previous method, due to the variety of its implementations, it’s one of the encoding mechanisms implemented among malware authors.

This method essentially converts binary data representation into a set of 64 characters.

There are several schemes for the different types of Base64 Encoding out there, but most contain the characters: [0-9,a-z,A-Z] and two additional characters, usually + and =, used for padding the string length to fit the format.

The conversion is done as follows: every 3 bytes of binary data are converted to 4 bytes of Base64 data.

Just to tickle your ears, the string ATT is represented by the following 3 bytes in hexadecimal: 0x41 0x54 0x54 (each character in ASCII is 8 bits which is one byte which is two characters in hex).

If we convert it to binary, we get this representation:

We’ll treat each 6 bits as a single character and convert them to decimal representation, meaning 010000 will be converted to 16 in decimal, 010101 will be converted to 21, and so on…

Now, if we go to the character table that represents our Base64 format:

Look for the indexes – 16, 21, 17, and 20, we will get the characters: QVRT accordingly.

Also here, the attacker has a wide range of possibilities for implementing Base64 Encoding.

He could, for instance, change the indexing table to have a different order, which will affect all the indexing and how the conversion is done.

Use Of Cryptography API Functions

Most encryption algorithms under this category are based on the fact that:

• Performing brute-forcing in an attempt to decrypt the source, given the complexity of the key and the encryption, simply isn’t possible (it would take decades – yes, decades…).
• The way the algorithm works is publicly accessible for inspection, but without the key, it’s impossible to decrypt the source.
• As mentioned earlier, using encryption algorithms in this category is usually easier to identify during the analysis of malware. This is reflected in the use of informative constants, aka ‘Consts’, importing libraries and functions, and others…

Let’s briefly go over the symmetric encryption algorithm – AES:

1. It is a symmetric algorithm meaning the same key is used for both encryption and decryption – we’ll remember this when analyzing the malware, aiming to find the key for performing the decoding.
2. The algorithm essentially takes 128 bits of content and turns it into 128 bits of cipher text (encrypted and unreadable content).
3. The key used for encryption can be in sizes of 128 bits, 192 bits, or 256 bits. Each size indicates the number of rounds that will be performed during the algorithm on the content we wish to encrypt (10 rounds, 12 rounds, or 14 rounds accordingly).
4. The algorithm essentially takes 16 bytes (128 bits), arranges them in a 4×4 grid, and performs operations of:XOR -> Byte Substitution -> Shifting Rows -> Mixing Columns -> Adding Round KeyWhere each such cycle (except for the XOR stage, which happens at the beginning and relies on a hidden key) is called a round.

Side Note:

In addition, a mechanism called ‘Key scheduling’ is also used in order not to use the same key in every round, but we won’t get into that right now.

Generally speaking, most modern encryption algorithms are based on Substitution Box and Permutation Box, where the previously mentioned Substitution Bytes stage is actually a Substitution Box.

And a Substitution Box is actually a lookup table where each byte is mapped to another byte based on a specific function.

For The Math Lovers Among Us➕

As a footnote for the math enthusiasts, an S-Box is a non-linear transformation of the form:

Meaning, each function fi in the field, receives n bits as input and returns fi for each xi in the following manner (meaning m bits).

Thus, a function in the field would be of the form:

Where for each f1,f2,…,fm such: fi is actually a Boolean function of the form:

Meaning, each such Boolean function fi returns us 0 or 1. Therefore, the final output we get will be m bits, or more precisely an m-ordered form of the sort:

Thus, in essence, the output of the Substitution Box stage for each byte is received.

In our case, since we talked about AES, the n parameter represents 128 bits. The whole mapping process we presented is actually one round (as mentioned earlier) and according to the algorithm, a number of such rounds will be performed.

Here comes the importance of the secret key mentioned earlier.

The key is divided into several “chunks” where each chunk essentially undergoes XOR with the output we received after each round we did – if we hadn’t done that, we could simply decrypt the algorithm by performing the operations we did in reverse order (given access to the sequence of operations themselves and without needing the key) – this stage is called key scheduling.

We won’t get into the Permutation Box stage in this article

The Data Decoding Checklist📈

Alright, so after “bashing you with some theory” (and a bit of math), we’ll dive into a partial analysis of a malware sample containing some of the principles presented here.

We’ll see that it is actually based on a custom encoding approach – it uses XOR encryption, Base64, and also AES encryption in the unique way it chose to implement.

In analyzing encrypted malware, we need to remember that we must identify the following three key principles:

1. Identify the type of encoding the malware uses (if at all), among them: AES, Base64, XOR Encoding, and others…
2. In case there is use of encoding, what are the functions aimed at performing the encoding.
3. How can we use these functions, the type of encoding and/or the memory addresses where the encoding occurs, to perform the decoding (deciphering) step.

Let’s Start Reversing😎

We started with the Basic Static Analysis stage and saw the existence of the following strings in the data segment of the malware:

CDEFGHIJKLMNOPQRSTUVWXYZABcdefghijklmnopqrstuvwxyzab0123456789+/

This is a Base64 index table (note the change in the order of characters in the index table as mentioned earlier) used for some sort of encoding, the malware will perform.

In addition, to a number of interesting functions: WriteFile, CreateThread, we can see several error message strings: Data not multiple of block size, Incorrect key length, and Empty key.

We can guess that it refers to some kind of encryption where the size of the block and the length of the key are important.

We also see a few strings are displayed as gibberish, which indeed confirms that the malware uses some form of encryption.

During the Basic Dynamic Analysis phase, we observed the malware attempting to communicate outward through port: 8910 to a domain we identified in the previous stage.

Subsequently, we see a cmd.exe window opening, simply waiting – we can infer that the malware might be trying to execute a reverse shell, where commands are actually sent to the malware on the previously seen port, for local execution at the endpoint.

A Little Side Note

We can use Netcat to set up a listener on port 8910 and use ApateDNS or INetSim to make the malware redirect all the requests it sends to the suspicious domain, to arrive at 127.0.0.1.

---

We can see that the malware tries to run commands like dir and we can see a lot of encrypted data received through the created connection.

Using PEiD, specifically with the KANAL Plugin, we can see memory addresses identified with constants and high entropy levels (in a word, entropy indicates memory addresses where the data is more random than usual, pointing to some form of implemented encryption).

Additionally, we can see the scan detected the use of an S-box structure implemented in various encryption algorithms (displayed as [S] and [S-inv] in the screenshot):

RIJNDAEL is another name for the encryption algorithm – AES – confirming our initial hypothesis about the use of some cryptographic algorithm.

Continue to analyze the addresses in memory characterized by high entropy, we notice that at address 0x004120A4, we find the Base64 index table we saw before.

We can conclude that the malware uses Base64 encoding with a custom index table and AES encryption.

Running a search for all the XOR commands existing in our assembly code, we see that there are quite a few:

We’ll ignore all commands that belong to functions related to Windows API libraries and commands where register cleanup is performed, namely commands in the form: xor eax,eax.

We’ll perform cross-reference to the addresses remaining after the filtering with the addresses we found before in the entropy scan and discover that there are 6 main functions containing these commands.

Here is an example code taken from one of these functions:

If we start to delve into the details of each assembly instruction in each of these functions, we will waste a lot of time – which is exactly what the malware writer would want us to do.

We’ll need to understand which function calls, whether they call one another (spoiler: they do) and generally understand the big picture of what is happening here.

Based on the fact that we expect to see some reference to AES encryption, we will try to understand how the XOR functions we found are related to this.

After some cross-referencing to the XOR functions, we discover that the main function is the one that calls the function whose name we changed to be xor_func1:

Since we anticipate references to AES encryption, the observant will notice that the offset of the string, which the function receives as a parameter, not only resembles a key but also comprises 16 characters.

These 16 characters, corresponding to 16 bytes (with each ASCII character representing one byte, totaling 128 bits), immediately suggest that we are dealing with AES-128 encryption, which is based on a 16-byte-long key.

Return To The Bigger Picture🌄

We saw earlier, a call to the function xor_func1 which receives a number of parameters, among them a string that looks like an AES key.

After that, we saw API functions calls to WSAStartup and WSASocketA whose purpose is to perform initialization for a network connection.

Then, the following code segment arrives:

Several additional calls are made whose purposes are:

   1. Performing DNS resolving for the domain passed as a parameter
   2. Converting the port (displayed as 8910) from little endian to big endian for the network convention during communication
   3. Performing connect to start the communication
   4. And calling another function *sub_4015B7*.

Starting to analyze the function *sub_4015B7*, we discover several interesting things:

   5. The malware creates a process of *cmd.exe*:

6. Two threads are created (we’ll discover later that each is intended for a different purpose):

7. Later on, we see a creation and linking of pipes to standard output, input, error as is commonly done when creating communication between standard input to output:

Given the fact that we found a Base64 index table, we can find the location in memory where it is saved (in the data segment of course) and perform cross-reference to see where it is used.

We will discover that it is referenced in the function sub_40103F which is called from sub_401082 and this in turn contains a nested loop.

Finally, we discover that the function sub_401082 is passed as a parameter to one of the threads we saw earlier (at the start address that the thread will run).

Importance Of Understanding The Chronological Order🔎

If we delve a bit into the code and the important addresses we saw, we will discover that:

1. There is a function that performs the decoding stage based on the Base64 index table called right after the call to CreateProcess that creates cmd.exe. Then, a socket connection is created with the calls (WSAStartup, WSASocketA, and additional functions from the library Ws2_32.dll) to create communication to the malicious domain.
2. A pipe is created that will pass the input/output from the created socket, to cmd.exe.
3. Analysis of the calls related to AES encryption reveals that the function responsible for performing the AES encryption is passed as a parameter to the second thread we saw, and this happens after the discussed process – cmd.exe has already started running.
4. Finally, we witness a call to the API function WriteFile in order to write to the cmd console – which in turn will perform a redirect of the output back through the created socket, out to the network.

When dealing with data encoding mechanisms, understanding the chronological order of the various encoding and decoding components is crucial. That’s the key to successfully decoding the encoded data!

What Have We Seen😎

• The malware we saw created a reverse shell, where the data it sends back to the C2 server undergoes encryption.
• In terms of the chronological order and the big picture, all the commands sent from the C2 to the reverse shell are sent encoded using Base64 and undergo decoding using the custom index table we found.
• Diving deeper, performing analysis and cross-referencing, we found out that all the responses sent back to the C2 server undergo AES encryption using the key:ijklmnopqrstuvwxright before being sent back.
• We reviewed the way AES encryption works: how does it work, what is it based on, a bit of mathematical explanation, and how it is displayed in the assembly code of the malware sample.
• Most importantly , we saw what the workflow we want to implement when investigating malware that use encryptions of various kinds:

1. We want to identify what kind of encryption/encoding the malware is using.
2. Discover which functions perform the encryption/encoding and which additional memory addresses are related to the process (for example, strings in the data segment).
3. How to perform pivoting around an address/function or any other piece of information we found in order to advance with understanding the full functionality of the malware.

What Haven’t We Seen (And Probably Will)👀

Last but not least, we did not manage to present here the *Decoding stage itself*.
The decoding can be performed in several ways, the main ones are:

1. Manual Decoding – in this approach, we identify the functions that perform the decoding, run the malware in a debugger, and see the strings decoded in real-time. If we fail to identify these functions, or if the malware does not perform decoding at any stage during the run – we’re out of luck…

2. Decoding Using Coding – we can write a script that will perform the decoding for us.

With that being said, we’ll need to give it all the necessary information for performing the decoding.

For example, if it’s Base64, we will need to give it the index table, if it’s AES or DES, we will need to give it the password (which we will have to extract from the assembly code) and the mode of encryption (for instance, AES contains several modes: CBC or EBC, etc…) – in other words, there’s quite a bit of data that we first need to extract from the assembly code, so that our code can run and perform what we command it to.

3. Instrumentation – Once we identify the functions responsible for performing the decoding and encoding and the parameters they receive, we can change the flow of the program’s execution so that it will perform the decoding for us – with the data we provide it.

We achieve this using plugins we write for OllyDbg, ImmDbg,x32/64Dbg that will perform modifications to the values of the registers and to memory addresses in order to perform the decoding for us.

This requires a fairly deep understanding of what happens behind the scenes, to allocate buffers from which we will write and read, changing the value of the EIP register (and other registers accordingly), and more…

It’s not the easiest to implement but once you understand it deeply, it can be applied to a significant number of malware families out there – definitely a skill worth acquiring for the long run.

A Side Note📃

We can implement this partially in the debugger itself.

In cases the encryption used by the malware is reversible (the same algorithm for encryption and decoding) we can simply give the function that performs the encoding, the encoded data and receive it decoded at the end of its run…

What’s Next😏

Since we already discussed the major objectives of malware, the vastly used Process Injection techniques and even got a glimpse of some User-Space Rootkit, the next step is exploring one of the most important aspects of malware – Network Traffic!

Malware have different purposes, but all of them are made to infiltrate data in one way or another.

Thus, we need to be able to research and analyze network traffic.

BUT, before we dive to the world of Network Traffic Analysis, we’ll do a short detour.

As you sure have noticed, reverse engineering malware requires some understanding of Low-Level knowledge and OS Internals.

Since this aspect is so important, we’ll review the difference between the main CPU architecture types – RISC and CISC. How does they differ from one another, and how does it affect the way malware are written on each architecture.

Once again, keep learning and Have Fun Reversing!

References

1. The following resources are taken from an amazing book called: Practical Malware Analysis By Michael Sikorski and Andrew Honig
   • The Malware Sample
   • The graph chart with the XOR and AES functions call image
   • The Base64 Index Table example image
   • The XOR initial explanatory image
   • The unsuccessful vs successful base64 decoding’ image
2. All other resources were generated through the analysis process

https://8ksec.io/dissecting-windows-malware-series-understanding-cryptography-and-data-encoding-part-3