Skip to content

Fix precision loss in compareDoubleInt/compareDoubleUint for values outside float64 safe integer range#1328

Open
XananasX7 wants to merge 2 commits into
google:masterfrom
XananasX7:fix/compareDoubleInt-precision
Open

Fix precision loss in compareDoubleInt/compareDoubleUint for values outside float64 safe integer range#1328
XananasX7 wants to merge 2 commits into
google:masterfrom
XananasX7:fix/compareDoubleInt-precision

Conversation

@XananasX7

Copy link
Copy Markdown

Summary

Fix precision loss in compareDoubleInt and compareDoubleUint when comparing double with int/uint values outside the safe float64 integer range (> 2^53).

Problem

The current implementation casts Int (int64) or Uint (uint64) to float64 before comparison:

func compareDoubleInt(d Double, i Int) Int {
    // ...bounds checks...
    return compareDouble(d, Double(i))  // Double(i) = float64(i): loses precision!
}

IEEE 754 float64 has a 53-bit mantissa. Any integer with absolute value greater than 2^53 = 9007199254740992 cannot be represented exactly as float64. This means two distinct int64 values can map to the same float64, causing the comparison to incorrectly return equal.

Demonstrated bug

// int(9007199254740993) == double(9007199254740992.0) evaluates to TRUE
// but these are different numbers — off by 1
result := compareDoubleInt(Double(9007199254740992), Int(9007199254740993))
// result == 0 (equal) — WRONG

Other affected values:

  • int(100000000000000001) == double(1e17)true (should be false)
  • int(1000000000000000001) == double(1e18)true (should be false)

Impact

CEL is used as the expression engine in GCP IAM Conditions, Firebase Security Rules, and Kubernetes admission webhooks. A policy rule that compares an integer field to a double literal near these boundary values can produce incorrect authorization decisions.

Fix

Use math/big.Float for exact comparison, eliminating the int→float64 precision loss:

func compareDoubleInt(d Double, i Int) Int {
    if d < math.MinInt64 { return IntNegOne }
    if d > math.MaxInt64 { return IntOne }
    bf := new(big.Float).SetFloat64(float64(d))
    bi := new(big.Float).SetInt64(int64(i))
    return Int(bf.Cmp(bi))
}

The same fix is applied to compareDoubleUint.

The fast-path bounds checks (< MinInt64, > MaxInt64) remain unchanged and avoid the big.Float allocation for values clearly out of range. The big.Float path is only reached for values in the representable int64 range where precision matters.

Testing

All existing comparison tests continue to pass. The previously incorrect cases now return the correct result:

int(9007199254740993) == double(9007199254740992.0) → false ✅
int(100000000000000001) == double(1e17)             → false ✅
int(1000000000000000001) == double(1e18)            → false ✅

@TristonianJones

Copy link
Copy Markdown
Collaborator

@jnthntatum @l46kok @jcking What are your thoughts on switching to BigInt, BigFloat under the covers for comparisons like this? If we were to make such a change we'd need it to be consistent across all stacks.

@jcking

jcking commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Isn't this the same problem as #1317?

I'd much rather not switch to BigFloat/BigInt, its a pretty heavy weight dep in other stacks and typically performs memory allocation. I am pretty sure you can implement this correctly without using those.

@TristonianJones

Copy link
Copy Markdown
Collaborator

@jcking it's mostly the same, but with a wider type representation. The way in which doubles are rounded depends on compiler flags and platform settings, so I'm not sure if switching to a wider representation above int53 precision would avoid other incongruities relating to floating point. That's my main thought.

@jcking

jcking commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

Rounding is controlled by IEEE 754 behavior. We should just be able to brute force it with:

func compare(a int64, b float64) Value {
  if math.IsInf(b) {
    if b < 0.0 {
      return 1;
    }
    return -1;
  }
  if math.IsNaN(b) {
    // Um...error?
  }
  // Round toward zero to nearest integer.
  c := math.Trunc(b)
  d := int64(c)
  // Compare non-fractional parts. If they are diffferent the fractional part doesn't matter.
  if a < d {
    return -1
  }
  if a > d {
    return 1
  }
  // Check the fractional part by removing the non-fractional part via subtraction.
  b -= c
  if c < 0 {
    return -1
  }
  if c > 0 {
    return 1
  }
  return 0
}

That should get pretty close. we would have to check the minimum representable value before zero and infinity and see if they hold. But that can be peeked at as well by force casting to int and back and comparing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants