Skip to content

CBORGenerator.Feature.WRITE_MINIMAL_INTS does not write most compact form for all integers #201

@yawkat

Description

@yawkat

Since cbor ints have the sign bit stored separately, 4-byte integers have an effective range of -0x100000000L to 0xffffffffL inclusive both. 0xffffffff can be encoded as positive 0xffffffff and -0x100000000 can be encoded as negative 0xffffffff (since 0x100000000 = -1 - 0xffffffff).

Right now, when writing long values with minimal int support enabled, the generator only goes down to the 4-byte representation when the long is actually within range for int, even though more numbers than that can be represented as long. This is simple to fix.

There are also other changes the protocol allows for. The largest integers representable by pos/neg 8-byte ints are never generated by jackson, since the API only accepts longs. Changing this requires adding new API, either one that accepts a signum parameter and a long magnitude, which is unfortunately fairly low level, or an API that accepts a BigInteger, which may be more computationally expensive.

Finally, the protocol allows for encoding integers as floats where that representation is shorter. This is mostly the case for large powers of two that could be represented as 16 or 32 bit floats instead of 32 or 64 bit integers. I have implemented this for an unreleased project, and this is the kotlin implementation:

    inline fun writeIntegerAsFloat(
            value: ULong,
            sign: Boolean,
            writeHeader: (ItemHeader) -> Unit,
            writeDirectPayload: (ItemHeader, ULong) -> Unit
    ): Boolean {
        if (value > 0xffff.toULong()) {
            val exponent = 63 - java.lang.Long.numberOfLeadingZeros(value.toLong())
            assert(exponent >= 16)
            val fractionLength = exponent - java.lang.Long.numberOfTrailingZeros(value.toLong())
            if (exponent <= 15 && fractionLength < 11) {
                val withBias = exponent + 15
                val fraction = (value shr (exponent - 10)).toInt() and 0x3f
                val float = (withBias shl 10) or fraction or (if (sign) 0x8000 else 0)
                val header = ItemHeader.SimpleDataType.float(2)
                writeHeader(header)
                writeDirectPayload(header, float.toULong())
                return true
            } else if (value >= 0xffffffff.toULong() && exponent <= 127 && fractionLength < 24) {
                var float = value.toFloat().toBits()
                if (sign) float = -float
                val header = ItemHeader.SimpleDataType.float(4)
                writeHeader(header)
                writeDirectPayload(header, float.toULong())
                return true
            } else {
                return false
            }
        } else {
            return false
        }
    }

The value of this is limited (2 to 4 bytes saved in some special cases) so I'm not sure if this code is really worth adding and maintaining, especially since 16 bit floats are annoying to deal with.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions