Add support for in-memory decompression #920

Sure thing! Obviously I'm not a UPX expert, so if my understanding is off, please feel free to correct me.

Sorry if I wasn't clear. To clarify, I'm not multiplexing multiple streams of packed binaries, I'm processing them each independently from their own byte streams.

Let's say I have 100 packed binaries and I want to run some analytics on the unpacked versions of these binaries and then run some additional processing based on the results of what I find in the unpacked versions. The way UPX is currently written relies heavily on command line processing, so I'd like to avoid running a subprocess that passes data to the UPX binary.

My current approach, with the forked copy I have, uses Python bindings. On the Python side, I read in each file as a binary and pass it to a custom pybind C++ function that process the decompressed portion only. I do this by using a custom pre-processor directive, MEMORY_INPUT, which is used in the file.cpp and file.h files to do the following:

1. Create a memory file descriptor, which replaces the file descriptor that is used for on-disk files when processed from the command line. So I include two functions for custom file descriptors; one for reading from bytes provided from the Python binding and the other for writing the decompressed bytes to memory so I that I can give them back to Python:

// For writing to a new OutputFile in memory only
int create_memfd(const char* name)

And this one:

// For reading from a file descriptor bytes that are in memory and not from an on-disk operation
int create_memfd(const char* name, const uint9_t* data, size_t size)

My pre-processor directive wraps those two functions in file.cpp.

2. For the InputFile object, I also override the sopen call by wrapping it too in the MEMORY_INPUT directive. The call takes the following:

void InputFile::sopen(const char *name, const uint8_t* data, size_t length)

Within this function it basically takes on the same values as the original call except I do:

_flags = O_BINARY;
_data = data;
_fd = create_memfd(name, _data, _length);

3. For the OutputFile object, I override the sopen call as well, with the following differences that are wrapped in my MEMORY_INPUT directive:

void OutputFile::sopen(const char *name, size_t length

Changes:

_flags = O_CREAT;
_fd = create_memfd(name);

4. I also provide my own custom dump method using the file descriptors created when wrapped by the directive

In addition to the above, I also bypass the calls to the gui processing defined in the packer.cpp file by simply wrapping those in an ifndef MEMORY_INPUT directive.

The only other modification I had to make was to the p_lx_elf.cpp file, specifically to the PackLinuxElf64::checkEhdr call because the uninitialized state osabi0 (which comes from the command line) was causing the decompression to behave inconsistently. I don't completely understand this section, but by providing a default option state for it from my binding and defining an #define ELFOSABI_SYSV 0 to check against osabi0, I was able to get consistent decompression results. I can get into more details about this if necessary.

Other than all that, I made some changes to the CMakeLists.txt file so I can build a custom .so for my Python bindings, but I think that's out-of-scope for this discussion.

The way UPX is currently written relies heavily on command line processing, so I'd like to avoid running a subprocess that passes data to the UPX binary.

Yes, UPX takes arguments from the command line. But UPX can process multiple files using only one process:

$ upx --help
Usage: upx [-123456789dlthVL] [-qvfk] [-o file] file..

The ellipsis in file... means that multiple instances of file [separated by whitespace] can appear. For example:

# Copy some execuables into a temporary directory.
$ cd /tmp; cp /bin/date date; cp /usr/bin/ls ls; cp /usr/bin/who who
# Process all of the copied execuables sequentially using only one process.
$ upx date ls who
        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    110952 ->     51112   46.07%   linux/amd64   date                          
    145520 ->     67284   46.24%   linux/amd64   ls                            
     61600 ->     29120   47.27%   linux/amd64   who                           
   --------------------   ------   -----------   -----------
    318072 ->    147516   46.38%                 [ 3 files ]

Packed 3 files.
$ 

Ah, I didn't consider UPX as a derivative user of another license with restrictions.

Thank you very much for the ZLIB suggestion, I will look into that!