Skip to content

feat: add support for xint64xx functions. - #796

Open
plusvic wants to merge 56 commits into
mainfrom
xint64xx-support
Open

plusvic wants to merge 56 commits into
mainfrom
xint64xx-support

Conversation

@plusvic

@plusvic plusvic commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

Adds support for 64-bit integer reading functions (int64, int64be, uint64, uint64be) and introduces 64-bit unsigned integer semantics across the parser, type system, IR, and WASM code generator.

Key Changes

Built-in 64-bit read functions

  • Implemented int64(offset), int64be(offset), uint64(offset), and uint64be(offset) for reading 64-bit signed and unsigned integers (little-endian and big-endian) from scanned data.

Unsigned 64-bit integer support

  • Literals: Integer literals that exceed i64::MAX (0x7FFFFFFFFFFFFFFF) up to u64::MAX (0xFFFFFFFFFFFFFFFF) are now parsed and represented as 64-bit unsigned integers instead of being rejected as out of range.
  • Comparisons (==, !=, <, <=, >, >=): Comparisons between signed and unsigned integers are mathematically exact at both compile time and runtime (e.g., 0xFFFFFFFFFFFFFFFF > -1 is true, and an unsigned integer is never equal to a negative signed integer).
  • Arithmetic & bitwise operations (+, -, *, \, %, &, |, ^):
    • Produce an unsigned result when at least one operand is unsigned and the remaining operands are either unsigned or non-negative constants; otherwise, the result is signed.
    • Added constant folding for integer division (\) and remainder (%), and guarded against WASM runtime traps when dividing i64::MIN by -1 (evaluating to undefined instead).
  • Shift operations (<<, >>): Preserve the signedness of the left operand, using logical right shifts (i64.shr_u) for unsigned integers and arithmetic right shifts (i64.shr_s) for signed integers.
  • Unary operations (-, ~):
    • Unary minus (-) always produces a signed integer; bitwise NOT (~) preserves the operand's signedness.
    • Added the unsigned_unary_op compiler warning when - or ~ is applied to an unsigned integer.
  • Compile-time range checks & diagnostics:
    • Generalized IntegerConstraint::Range to i128 bounds so both signed (i64) and unsigned (u64) ranges can be represented and checked for unsatisfiable_expr / invariant_expr warnings.
    • Reject unsigned constants exceeding i64::MAX in range bounds and quantifiers where negative or out-of-range i64 values are invalid.

prosperritty and others added 30 commits October 7, 2026 10:33
…ition. (#792)

One problem with this highlighting approach is that it sometimes doesn't disappear even after the cursor is no longer over the rule's name. The second problem is that when a relatively large rule is highlighted, it can be quite annoying. Instead of highlighting the entire rule definition, I suggest highlighting only the name, if that is acceptable. It is what I've changed in this PR.


---------

Co-authored-by: Victor M. Alvarez <vmalvarez@virustotal.com>
The text explaining how to install `yara-x` using `brew` in macOS was in the wrong section of the documentation.
Also generalize the extraction of header constraints from integer reads so
that it supports reads of any width, including 8-byte reads.

Part of #26.
Integers are still stored as `i64`, but `TypeValue::Integer` now has an
`unsigned` flag indicating that the value must be interpreted as the bit
pattern of an `u64`. Signed and unsigned integers share `Type::Integer`,
so existing type checks accept both, and only operations where signedness
matters need to take it into account.

Nothing produces unsigned integers yet, so there are no user-visible
changes. The serialization version is bumped because `TypeValue` is part
of the serialized globals.

Part of #26.
Integer literals that don't fit in a signed 64-bit integer but fit in an
unsigned one (e.g: `0xFFFFFFFFFFFFFFFF`) are now unsigned integers.

* Comparisons between signed and unsigned integers are mathematically
  exact (e.g: `0xFFFFFFFFFFFFFFFF > -1` is always true).
* Arithmetic and bitwise operations produce an unsigned result only when
  every operand is either unsigned or a non-negative constant, otherwise
  the result is signed. Division and modulo use unsigned instructions
  for unsigned results.
* Shifts take the signedness of the left operand. Right shifts of
  unsigned integers are logical.
* Unary minus always produces a signed result, bitwise not preserves
  signedness. Both emit the new `unsigned_unary_op` warning when applied
  to unsigned integers.
* Constant folding of integer arithmetic is now exact (previously it was
  performed using floats, losing precision for large values), and
  results that don't fit in the result type are reported as errors.

Part of #26.
These functions read a 64-bit unsigned integer from the scanned data,
in little-endian and big-endian respectively. The result is an unsigned
integer, so values greater than `0x7FFFFFFFFFFFFFFF` are never negative.

Functions exported to WASM can now return `u64`, which is represented
as `i:U` in mangled names. Header constraints are also extracted from
comparisons involving `uint64` and `uint64be`.

Part of #26.
Module fields of type `uint64` are now unsigned integers, instead of
being reinterpreted as signed integers. This means that values greater
than `0x7FFFFFFFFFFFFFFF` are no longer negative, and that comparisons
and arithmetic follow the rules for unsigned integers.

Repeated `uint64` fields are represented by the new
`Array::UnsignedIntegers` variant. Map keys of type `uint64` still use
signed integers, but lookups with unsigned keys work as expected
because keys are compared by their bit patterns.

Part of #26.
Instead of an `i64` value plus an `unsigned` flag, `LiteralInteger::value` is now an `IntegerValue` enum with `Signed(i64)` and `Unsigned(u64)` variants.

Part of #26.
They are now `TypeValue::const_signed_integer_from` and `TypeValue::const_unsigned_integer_from`, for coherency.

Part of #26.
They are now `TypeValue::unknown_signed_integer` and `TypeValue::unknown_unsigned_integer`, for coherency.

Part of #26.
…eValue`.

`unknown_integer_with_constraints`, `var_integer_from`, `var_unsigned_from` and `try_as_unsigned` are now `unknown_signed_integer_with_constraints`, `var_signed_integer_from`, `var_unsigned_integer_from` and `try_as_unsigned_integer`, for coherency.

Part of #26.
…flag.

Unsigned integers are now represented by their own `TypeValue` variant,
holding a `Value<u64>`, instead of a `TypeValue::Integer` with an
`unsigned` flag and the `i64` bit pattern of the value. Both variants
share the same `Type::Integer` type.

Code that needs the raw 64 bits of an integer, regardless of its
signedness (e.g: WASM lookups and constant folding of bitwise
operations), uses the new `try_as_integer_bits` and `as_integer_bits`
methods.

Part of #26.
Also move `TypeValue::UnsignedInteger` next to it. This changes the discriminants of the variants that follow, and therefore the hashes in the IR golden files.

Part of #26.
`IntegerConstraint::Range` now uses `i128` bounds, which can represent
ranges for both signed and unsigned integers, and
`TypeValue::UnsignedInteger` has a `constraints` field, like
`TypeValue::SignedInteger`.

Constraints are checked when a constant integer, signed or unsigned, is
compared with a constrained integer of any signedness. Mangled names
can also combine the `U` and `R` constraints (e.g: `i:U:R0:255`).

Part of #26.
Having two `TypeValue` variants (`SignedInteger` and `UnsignedInteger`)
with the largest size prevented the compiler from storing the enum
discriminant in a niche, growing `Expr` from 24 to 28 bytes in 32-bit
Linux. Integer constraints are now stored as `Option<Box<[_]>>`
instead of `Option<Vec<_>>`, which also reduces `Expr` from 48 to 40
bytes in 64-bit platforms.

Part of #26.
Renames `IntegerValue` enum variants from `Signed` to `I64` and `Unsigned` to `U64`. This explicitly indicates the underlying `i64` and `u64` primitive types, improving clarity and consistency in integer handling throughout the compiler's AST and IR stages.
…d LiteralInteger.

Remove TypeValue::SignedInteger, TypeValue::UnsignedInteger, and
ast::IntegerValue. Both TypeValue::Integer and ast::LiteralInteger now
store the integer bit pattern as an i64 along with an is_unsigned
boolean flag.
Ensure negative `i64` start values are explicitly clamped to `0` before
conversion to `usize`. This clarifies the intent for handling
out-of-bounds negative indices and improves robustness.
Rewrite the documentation section on signed and unsigned integers to provide a more detailed explanation of their behavior, especially when mixed in operations, and include illustrative examples.
Adds comprehensive documentation explaining how integer signedness is handled in YARA-X. Specifically clarifies why `uint8`, `uint16`, and `uint32` functions produce signed 64-bit integers with range constraints, rather than unsigned integers. This design prevents unexpected wrap-around on underflow and simplifies comparisons with other signed integers.

Updates the `wasm_export` macro to consistently represent `RangedInteger` as a signed type with a range, aligning with the detailed explanation provided in the documentation.
@codecov

codecov Bot commented Oct 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.06205% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.40%. Comparing base (2ff51e9) to head (37e9638).
⚠️ Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
lib/src/compiler/ir/mod.rs 93.36% 16 Missing ⚠️
lib/src/compiler/emit.rs 91.79% 11 Missing ⚠️
lib/src/compiler/ir/dfs.rs 84.61% 2 Missing ⚠️
lib/src/compiler/ir/ast2ir.rs 99.00% 1 Missing ⚠️
lib/src/types/mod.rs 99.34% 1 Missing ⚠️
lib/src/types/structure.rs 97.05% 1 Missing ⚠️
lib/src/variables.rs 88.88% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #796      +/-   ##
==========================================
+ Coverage   93.25%   93.40%   +0.14%     
==========================================
  Files         153      153              
  Lines       44178    44647     +469     
==========================================
+ Hits        41200    41701     +501     
+ Misses       2978     2946      -32     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This change aligns the internal representation and arithmetic behavior of
protobuf `uint32` fields with the documented design for `uint8`, `uint16`,
and `uint32` functions. These types are now treated as signed 64-bit
integers with a value range constraint of `[0, u32::MAX]`, preventing
unexpected wrap-around on underflow and simplifying comparisons with other
signed integers.
@plusvic
plusvic enabled auto-merge (squash) October 9, 2026 10:05
@plusvic
plusvic disabled auto-merge October 9, 2026 10:08
plusvic added 18 commits October 9, 2026 12:54
The runtime guard added for range bounds treated any negative bound as an
empty range. That is only correct for unsigned bounds greater than
i64::MAX, which have a negative bit pattern when interpreted as i64.
Signed lower bounds can legitimately be negative at runtime (e.g:
`filesize - 10` or `@a[1] - 4`), and on main the loop iterated over the
negative values. Restrict the check to unsigned bounds, while keeping the
explicit `upper_bound < lower_bound` check, which prevents the number of
iterations from wrapping around into a positive value.
The number of iterations required by a percentage quantifier is computed
as `ceil(n * percentage / 100)` in floating point and converted back to
an integer with `i64.trunc_f64_s`, which traps if the value doesn't fit
in an i64. Percentages are guaranteed to be in [0, 100] only when they
are constant, but with functions like `int64` and `uint64` a percentage
computed from the scanned data can easily overflow, and the trap made the
scanner panic.

Use the saturating conversion `i64.trunc_sat_f64_s` instead, which
produces i64::MIN or i64::MAX, values that are never reached. Also convert
unsigned percentages to float as unsigned integers, so that values greater
than i64::MAX are not treated as negative.
…d dotnet

Protobuf `uint64` fields are unsigned integers in YARA rules, which means
that subtractions that underflow wrap around to huge positive numbers
(e.g: `pe.number_of_exports - 1 < 10` is false when there are no
exports). This is a problem for fields that are counts, which never exceed
i64::MAX, and are frequently used in arithmetic expressions.

Count fields (`number_of_*`, `*_entries`) are now `uint32`, which in YARA
rules are signed integers constrained to the range [0, 4294967295]. These
values never wrap around, but they are still known to be non-negative, so
combining them with an unsigned `uint64` value produces an unsigned result.
Fields that can legitimately exceed i64::MAX, like addresses, offsets and
sizes, remain `uint64`.

The JSON/YAML output of module dumps is not affected, but the Rust types
of these fields in `yara_x::mods` change from `u64` to `u32`.
The [0, u32::MAX] constraint for protobuf `uint32` fields was attached only
when the field's value was unknown. Fields in proto3 modules always have
a value (they default to 0, even at compile time), so they lost the
constraint, and therefore weren't treated as non-negative integers. Attach
the constraint regardless of the field's value.
The `*.no-folding.ir` golden files were outdated, and the ones for
inputs 13 to 16 didn't exist. The error in `errors/161.in` and two
assertions in `unsigned_integers` depend on constant folding, so they
are now skipped when the feature is disabled.
When the whole workspace is tested, Cargo unifies the features of
`yara-x` across all workspace members. Some of them depend on `yara-x`
through the `[workspace.dependencies]` entry, which enables
`constant-folding`, `exact-atoms`, `fast-regexp` and
`generate-proto-code`, so the `no-default-features` job never tested
`yara-x` without those features.
This reverts commit 736b06c, which changed these fields to `uint32`.
That changed the type of the fields in the public `yara_x::mods` API
from `u64` to `u32`, which is a breaking change.

As a consequence, these fields are unsigned integers in YARA rules again.
…isfied.

Negative quantifiers computed at runtime can't be satisfied, but small
negative percentages were an exception. When `n * percentage` is in the
range (-100, 0), `max_count` was rounded to 0, which means `none`. For
instance, `for int8(0) % i in (0..9) : (false)` was true when `int8(0)`
was -1. Negative percentages are now replaced with i64::MAX, which
produces a `max_count` that is never reached.
… unsigned result.

The signedness of arithmetic and bitwise operations depended on the values
of the operands. A signed operand that was a non-negative constant adopted the
signedness of the unsigned operands, while any other signed operand made the
result signed. As the value of an expression like `4 \ 2` is known only when
constant folding is enabled, `(4 \ 2) * (0xFFFFFFFFFFFFFFFF \ 2)` was unsigned
with constant folding, and signed without it. Something similar happened with
loop variables: `i * x` was signed even if `i` could only take the value 2,
while `2 * x` was unsigned.

Now the result is unsigned if at least one of the operands is unsigned,
regardless of the remaining operands, which are interpreted as unsigned, like
in C. This means that negative values wrap around (e.g: `uint64(0) + (-1)` is
equivalent to `uint64(0) - 1`). The signedness depends only on the types of
the operands, so it doesn't change with constant folding.
…tegers.

The documentation said that the values returned by `uint8`, `uint16` and
`uint32` are signed integers, which looks inconsistent with `uint64` being
unsigned. These values are never negative, but they are promoted to signed
64-bit integers because all of them fit in one, like C promotes the integer
types narrower than `int` to `int`. This also keeps them compatible with YARA.
`uint64` values can't be promoted, which is why they are the only unsigned
integers.

The documentation now uses that explanation, and shows the difference between
`uint32(0) - 1 < 0` and `uint64(0) - 1 < 0`.
Since unsigned literals were introduced, a minus sign followed by an unsigned
literal other than 9223372036854775808 produced a unary minus applied to an
unsigned integer, so literals like `-9223372036854775809` or
`-18446744073709551615` were accepted and wrapped around (e.g:
`-18446744073709551615 == 1` was true), instead of being rejected as before.
Now these literals are invalid integers again.

Additionally, constant folding of unary minus wrapped around silently, so
`-(-9223372036854775808)` was equal to -9223372036854775808, while
`0 - (-9223372036854775808)` was an error. Now unary minus is folded using
exact arithmetic, like the other arithmetic operations, and produces an error
when the result doesn't fit in a signed integer. This also applies to
unsigned constants, like in `-(0xFFFFFFFFFFFFFFFF)`. When the operand is not
known at compile time the result still wraps around.
…bool` consistent.

Both functions now receive the compile context, the `ExprId` of the
expression being checked and the span where the warning is reported,
and they obtain the expression's type by themselves. `warn_if_unsigned`
additionally receives the unary operator (`-` or `~`), and the note
shown in the warning is now chosen from the operator instead of being
passed by every caller.

`warn_if_not_bool` was only used in `ast2ir.rs`, so it is now private.
`TypeValue::integer_range` returned `None` for constants, which is what
prevented `eq_check` from checking a constant against the range of the
other constant's type in comparisons like `0xFFFFFFFFFFFFFFFF == -1`.
Now `integer_range` depends only on the type, and `eq_check` performs
the range check only when one operand is constant and the other is not.

The new `test_17` in `warnings/no_warnings.in` makes sure that comparing
constants with different signedness doesn't raise `unsatisfiable_expr`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants