Skip to content

Conversation

@ghaerr
Copy link
Owner

@ghaerr ghaerr commented Apr 10, 2024

As discussed briefly in #1864 (comment), this PR adds a new program decomp that is run by a user to decompress an ELKS binary on the distribution disk so that it loads faster. This will be used with an upcoming enhancement that will produce ELKS distribution images with compressed executables for commands with infrequent use. For users that desire quicker loading, decomp can be run on a running ELKS system.

Usage: decomp <program> [<decompressed_program>]. Without a second argument, the executable is compressed in-place and replaced. Providing a second argument allows for testing to compare executable file sizes and load times, if desired. The size of each program section is displayed, along with the total.

Due to the distribution disks being almost entirely full, decomp is currently on the 1440k image. I hope to be able to get it on the 360k image with mostly compressed binaries.

@toncho11
Copy link
Contributor

toncho11 commented Apr 10, 2024

So the kernel will decompress the executable automatically in memory (because there is no space on the floppy images) and then execute it every time. But this will be only for certain commands? I mean ls should not be compressed.

If something go wrong with the decompression it will effectively disable a bunch of commands/programs.

I am also thinking that the space will always be limited ... even with compression we will eventually meet the same problem one day. Maybe for 0.8 there should be a few test programs the are compressed so that this feature is thoroughly tested.

Also HD images do not need compression.

@ghaerr
Copy link
Owner Author

ghaerr commented Apr 10, 2024

@toncho11, kernel decompression has been implemented for probably 3+ years now, it works well.

I added the feature years ago and one person tested it, saying it was slow on an 8088. The current elkscmd/Applications mechanism currently allows only for all or no programs to be compressed, so I stopped with the idea of fully compressed disk images. Now, with decomp, if a user wants to decompress ls, running decomp ls would uncompress it permanently on the distribution. Of course, there may be some commands that should not be compressed, the individual selection feature likely needs to be added before we should move to higher density distributions, and user testing should occur on real hardware. On QEMU, there is no discernible difference when running fully compressed distributions other than noticing that tons more space is available (~450k on 1440k floppies).

The idea would be to have a mechanism that allows for specifically setting which programs would be compressed for which disk image sizes. That has not been built yet.

Don't understimate the utility of having more space... the current executable compression facility typically reduces file sizes by an average of around 30%. This makes a huge difference for 360k and all other floppy types. We're talking about 20+ more programs per disk image, at least, rather than having the lower density disk images having very few usable programs. This is only made worse with the recent games being added - only one is available on 360k.

If something go wrong with the decompression it will effectively disable a bunch of commands/programs.

The feature is entirely transparent. If the disk has a bad sector, the program wouldn't run regardless of being compressed or not. All compressed executables will always decompress properly otherwise. The algorithm is actually extremely cool - it decompresses in place without using any extra memory, so it will also always work even on systems with very little memory. The time it takes to read the program from floppy is also decreased, since the executable is smaller. The only real issue is that very slow 8088 systems take possibly as much time to decompress as the time saved from reading from the floppy, except when the program is read from the kernel system buffers rather than disk.

A 360k compressed disk image can be created now by setting CONFIG_APPS_COMPRESS=y in .config or make menuconfig. Then the entire floppy will be compressed executables and should have ~100k free on 360k, rather than the 0k free now.

@toncho11
Copy link
Contributor

So each time a compressed executable is called the decompressed one is stored on the disk temporally?

@ghaerr
Copy link
Owner Author

ghaerr commented Apr 10, 2024

No, each time a compressed executable is executed, it is either fetched from the kernel system buffers, or from disk and loaded into memory. After that, the executable is decompressed in-place, in-memory, then executed. On slower 8088 systems, the actual in-memory decompression takes some time for larger executables like vi (59k). I'm not sure anyone measured the effects on smaller exes.

@ghaerr ghaerr merged commit 408fb3c into master Apr 11, 2024
@ghaerr ghaerr deleted the decomp branch April 11, 2024 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants