Improve ALP compression ratios for Float32 data#111627
Conversation
... to drastically reduce the number of exceptions, improving compression ratio
|
Workflow [PR], commit [47ee9f7] AI ReviewSummaryThis PR switches Findings❌ Blockers
Tests
Final Verdict
|
| T value_dec = value_float * ALPFloatTraits<T>::EXPONENTS[fraction] * ALPFloatTraits<T>::FRACTIONS[exponent]; | ||
| return value_dec; | ||
| /// It's important to keep two multiplication steps as float multiplication is not associative. | ||
| Float64 value_dec = static_cast<Float64>(value) * EXPONENTS[fraction] * FRACTIONS[exponent]; |
There was a problem hiding this comment.
This changes the Float32 ALP(STD) decode arithmetic for the existing version-1 bitstream, but the stream header still stays at ALP_CODEC_VERSION = 1. That means already-written parts can silently decode to different bits after upgrade: for example, under the old math -2000000.0f is a valid encoded value with (e=7, f=4, d=-1999999872) and round-trips exactly, while this decoder reconstructs -1999999.875f from the same payload. We need a new codec version (or another explicit format flag) so doDecompressData can keep the legacy Float32 path for old data and use the promoted Float64 path only for newly written streams.
There was a problem hiding this comment.
No need to keep legacy path on experimental feature
ALP(STD)compresses a floating-point column by turning each value into an integer. Valuenis scaled asd = round(n * 10^e * 10^-f), for some whole numberseandfs.t.f ≤ e. If scaling back asd * 10^f * 10^-ereproduces the exact bits ofn, the integer is stored bit-packed. Otherwisenis stored verbatim as an exception.In native
Float32arithmetic from ALP reference implementation, the scaling constantfl32(10^-e)is inexact enough that many decimal values (e.g.0.05f) fail the bit-exact round-trip verification and are stored as expensive exceptions (example). Promoting the scaling arithmetic toFloat64makes everyFloat32nearest to a decimal of up to 9 fractional digits round-trip bit-exactly. This doesn't have a noticeable performance impact, but quite a noticeable compression ratio change (i.e. 2x smallerl_discounton TPC-H SF1).Also simplifies the code now that the
Float32constant set is gone.Columns written by earlier versions may decode values 1 ulp away. This is intentional, the codec is experimental.
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):
The experimental
ALPcodec now performsFloat32scaling arithmetic inFloat64, which eliminates exceptions on decimal data and significantly improves compression ratios.