Cleanup Precision #3018
Cleanup Precision #3018
Conversation
| // Math.floor always rounds to next smaller number so negative values will be like -3.1 -> -4 | ||
| // +1 as we are using floor and in both positive and negative exponents we need +1 | ||
| final double max = Math.max(Math.abs(a), Math.abs(b)); | ||
| final double scaledSignificance = Math.pow(10, sigDigits * -1d) * max; |
There was a problem hiding this comment.
this pow is not needed. The old version does pow(log10(pow)) the new version just does pow(log10)
| } | ||
| final double diff = Math.abs(a - b); | ||
| // if one is infinite or if number overflow | ||
| if (Double.isInfinite(diff)) { |
There was a problem hiding this comment.
to handle number overflow
| * NaN, false if only one is NaN | ||
| */ | ||
| public static boolean equalRelativeSignificance(final double a, final double b, | ||
| final double relativeError) { |
There was a problem hiding this comment.
I checked the methods available in apache commons math Precision.
They have one that just uses relative value precision and maybe thats actually fine? Then we dont need all the fancy log10, floor, pow10. Maybe forcing it into the decimal system with pow10 is actually bad because we still get all the flaoting point inaccuracies like we want 10 but it may be 9.99999.
Only issue in apache version is that it returns false if both values are NaN.
So better to reimplement here.
to check default float accuracy we could either use 1E-5 or 5E-6 for checks
| public static boolean equalRelativeSignificance(final double a, final double b, | ||
| final double relativeError) { | ||
| // equal or both are NaN | ||
| if (Double.compare(a, b) == 0) { |
There was a problem hiding this comment.
a = positive and b = negative infinity will not be equal, or? they may cause number overflow later
There was a problem hiding this comment.
checked. This this comes down to max and a-b are both INFINITY and devided this is NaN and NaN is always false when compared to any other number
There was a problem hiding this comment.
so test result is correct
Cleaned up the changes that I tried in precision class.
I checked and Float.NaN and Float.POSITIVE_INFINITY are perfectly converted to Double so that no extra checks are required.
Double.compare(a, b) == 0 already covers the case where both values are NaN so I removed the extra checks.
Added a check for the Double.isInfinite(diff) to capture number overflow.