-
Notifications
You must be signed in to change notification settings - Fork 6
capstone, upx, fq: complete the reverse-engineering workbench #549
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
abd042b
capstone: package the disassembly framework 5.0.9
bryan-minimal 842309e
upx: package the executable packer 5.2.0
bryan-minimal d916168
fq: package the binary-format query tool 0.17.0
bryan-minimal 0c775c2
capstone, upx: record the aggregate licence, not the headline one
bryan-minimal File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| let { standaloneTest, Attrs, BuildSpec, Local, OutputBin, OutputData, OutputLib, Source, Test, .. } = import "minimal.ncl" in | ||
| let base = import "../base/build.ncl" in | ||
| let cmake = import "../cmake/build.ncl" in | ||
| let ninja = import "../ninja/build.ncl" in | ||
| let pkgconf = import "../pkgconf/build.ncl" in | ||
| let toolchain = import "../toolchain/build.ncl" in | ||
| let glibc = import "../glibc/build.ncl" in | ||
| let version = "5.0.9" in | ||
| { | ||
| name = "capstone", | ||
| build_deps = [ | ||
| { file = "build.sh" } | Local, | ||
| base, | ||
| cmake, | ||
| ninja, | ||
| pkgconf, | ||
| toolchain, | ||
| glibc, | ||
| { | ||
| url = "https://github.com/capstone-engine/capstone/archive/refs/tags/%{version}.tar.gz", | ||
| sha256 = "0619da31af08152600af95c481527ef6d756c0a8404fca7544a4fdf6dfc2c0f9", | ||
| extract = true, | ||
| strip_prefix = "capstone-%{version}", | ||
| } | Source, | ||
| ], | ||
| cmd = "./build.sh", | ||
| outputs = { | ||
| cstool = { glob = "usr/bin/cstool" } | OutputBin, | ||
| libs = { glob = "usr/lib/libcapstone.so*" } | OutputLib, | ||
| includes = { glob = "usr/include/**" } | OutputData, | ||
| pkgconfig = { glob = "usr/lib/pkgconfig/*.pc" } | OutputData, | ||
| cmake_files = { glob = "usr/lib/cmake/**" } | OutputData, | ||
| }, | ||
| runtime_deps = [glibc], | ||
| attrs = | ||
| { | ||
| upstream_version = version, | ||
| source_provenance = { category = 'GithubRepo, owner = "capstone-engine", repo = "capstone" }, | ||
| # Two licences, both permissive, and the tree ships both files: | ||
| # LICENSE.TXT BSD-3-Clause (COSEINC) | ||
| # LICENSE_LLVM.TXT University of Illinois/NCSA Open Source License, | ||
| # covering the arch decoders generated from LLVM's | ||
| # tables. This is the OLD "LLVM Release License" — | ||
| # capstone 5.x predates LLVM's relicensing, so it is | ||
| # NCSA here, NOT Apache-2.0 WITH LLVM-exception. | ||
| # Recorded as the aggregate rather than the headline licence, since | ||
| # NCSA's attribution clause travels with the binary too. | ||
| license_spdx = "(BSD-3-Clause AND NCSA)", | ||
| } | Attrs, | ||
| tests = { | ||
| smoketest = standaloneTest "/bin/cstool -v", | ||
|
|
||
| # DISASSEMBLE, don't just run --version. | ||
| # | ||
| # The failure this guards is a build that works and is quietly less | ||
| # capable. `CAPSTONE_ARCHITECTURE_DEFAULT` gates every per-arch | ||
| # `CAPSTONE_<ARCH>_SUPPORT` option, so changing one value drops whole | ||
| # architectures out of the library while cstool still builds, still runs, | ||
| # and still reports the same version. Nothing else here would notice: | ||
| # `enumerate bins` sees a binary, `output types valid` sees a .so, a | ||
| # smoke test sees a version string. | ||
| # | ||
| # So the assertions are real instructions on real bytes, including two | ||
| # fringe architectures — the ones a "slim it down" change removes first, | ||
| # and the ones an RE workbench misses last. | ||
| disassembles = | ||
| { | ||
| class = 'Standalone, | ||
| test_deps = [base], | ||
| cmds = [ | ||
| # x86-64: push rbp / mov rbp, rsp — the universal function prologue. | ||
| ["/bin/bash", "-c", "cstool x64 '554889e5' | grep -qi push && cstool x64 '554889e5' | grep -qi mov"], | ||
| # arm64: mul x1, x1, x2 — capstone's own test vector. | ||
| ["/bin/bash", "-c", "cstool arm64 '217c029b' | grep -qi mul"], | ||
| # 32-bit ARM: a branch-with-link. | ||
| ["/bin/bash", "-c", "cstool arm 'edffffeb' | grep -qi bl"], | ||
| # Fringe #1 — riscv64 (0x00000013 = nop). | ||
| ["/bin/bash", "-c", "cstool riscv64 '13000000' | grep -qiE 'nop|addi'"], | ||
| # Fringe #2 — ppc64, BIG-endian. `ppc64` alone is CS_MODE_LITTLE_ENDIAN | ||
| # in capstone 5 (cstool.c: "ppc64" => CS_MODE_64|CS_MODE_LITTLE_ENDIAN), | ||
| # and 7c0802a6 is the big-endian encoding of `mflr r0` — the mode | ||
| # name has to match the bytes or this decodes to garbage. Caught by | ||
| # running it: the first version used `ppc64` and failed. | ||
| ["/bin/bash", "-c", "cstool ppc64be '7c0802a6' | grep -qi mflr"], | ||
| # CONTROL: an architecture that does not exist must FAIL. Without | ||
| # this, every grep above could be matching an error message rather | ||
| # than disassembly, and the whole test would prove nothing. | ||
| ["/bin/bash", "-c", "! cstool no-such-arch 00 >/dev/null 2>&1"], | ||
| ], | ||
| } | Test, | ||
|
|
||
| # The shared library is the reason to package this rather than let each | ||
| # consumer vendor it. `BUILD_SHARED_LIBS` defaults to **OFF** upstream, so | ||
| # a stock build produces libcapstone.a only — and anything linking that | ||
| # absorbs capstone statically, making a capstone CVE invisible to pkgscan. | ||
| # That default is one line in build.sh away from silently coming back. | ||
| ships_shared_lib = | ||
| { | ||
| class = 'Standalone, | ||
| test_deps = [base, pkgconf], | ||
| cmds = [ | ||
| ["/bin/bash", "-c", "test -e /usr/lib/libcapstone.so"], | ||
| # A real versioned SONAME, not just the dev symlink. | ||
| ["/bin/bash", "-c", "ls /usr/lib/libcapstone.so.* >/dev/null 2>&1"], | ||
| # ...and no static archive, so nobody links it by accident. | ||
| ["/bin/bash", "-c", "! test -e /usr/lib/libcapstone.a"], | ||
| # pkg-config must resolve, or a downstream build quietly falls back | ||
| # to its own vendored capstone. | ||
| ["/bin/bash", "-c", "PKG_CONFIG_PATH=/usr/lib/pkgconfig pkg-config --exists capstone"], | ||
| ], | ||
| } | Test, | ||
| }, | ||
| } | BuildSpec | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,40 @@ | ||
| #!/bin/sh | ||
| set -eu | ||
|
|
||
| # Reproducibility flags (see AGENTS.md). | ||
| export CFLAGS="${CFLAGS:-} -ffile-prefix-map=$(pwd)=/builddir -gno-record-gcc-switches" | ||
| export CXXFLAGS="$CFLAGS" | ||
| export LDFLAGS="${LDFLAGS:-} -Wl,--build-id=none" | ||
| export ARFLAGS=Drc | ||
|
|
||
| # BUILD_SHARED_LIBS defaults to OFF and BUILD_STATIC_LIBS to ON upstream — the | ||
| # opposite of what a distro wants. Left alone, this package would ship | ||
| # libcapstone.a and every consumer would absorb capstone statically, which puts | ||
| # a capstone CVE beyond pkgscan's reach: nothing in the consumer's tree would | ||
| # name capstone at all. So shared on, static off, deliberately. | ||
| # | ||
| # CMAKE_INSTALL_LIBDIR=lib keeps the installed .pc and cmake files pointing at | ||
| # usr/lib rather than the GNUInstallDirs 64-bit default lib64. | ||
| # | ||
| # CAPSTONE_ARCHITECTURE_DEFAULT=ON is the upstream default and is set here | ||
| # explicitly because it is load-bearing: it gates every per-architecture | ||
| # CAPSTONE_<ARCH>_SUPPORT option at once, and turning it off yields a working | ||
| # cstool that silently cannot disassemble whole architectures. The | ||
| # `disassembles` test pins the consequence rather than trusting this line. | ||
| # | ||
| # Tests off: upstream's suite needs its own fixtures and adds build time; the | ||
| # standalone tests in build.ncl assert the properties we actually care about. | ||
| cmake -S . -B build -G Ninja \ | ||
| -DCMAKE_INSTALL_PREFIX=/usr \ | ||
| -DCMAKE_INSTALL_LIBDIR=lib \ | ||
| -DCMAKE_BUILD_TYPE=Release \ | ||
| -DBUILD_SHARED_LIBS=ON \ | ||
| -DBUILD_STATIC_LIBS=OFF \ | ||
| -DCAPSTONE_BUILD_CSTOOL=ON \ | ||
| -DCAPSTONE_BUILD_TESTS=OFF \ | ||
| -DCAPSTONE_BUILD_CSTEST=OFF \ | ||
| -DCAPSTONE_ARCHITECTURE_DEFAULT=ON \ | ||
| -DCAPSTONE_X86_REDUCE=OFF | ||
|
|
||
| cmake --build build -j"$(nproc)" | ||
| DESTDIR="$OUTPUT_DIR" cmake --install build |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| let { standaloneTest, Attrs, BuildSpec, Local, Needs, OutputBin, Source, Test, .. } = import "minimal.ncl" in | ||
| let base = import "../base/build.ncl" in | ||
| let go = import "../go/build.ncl" in | ||
| let toolchain = import "../toolchain/build.ncl" in | ||
| let version = "0.17.0" in | ||
| { | ||
| name = "fq", | ||
| build_deps = [ | ||
| { file = "build.sh" } | Local, | ||
| base, | ||
| go, | ||
| toolchain, | ||
| { | ||
| url = "https://github.com/wader/fq/archive/refs/tags/v%{version}.tar.gz", | ||
| sha256 = "c5658b2bc635a1d344c64e37d7311157f0fc4b20cc3cfa4d09bdd2f023692d57", | ||
| extract = true, | ||
| strip_prefix = "fq-%{version}", | ||
| } | Source, | ||
| ], | ||
| needs = | ||
| { | ||
| dns = {}, | ||
| internet = {}, | ||
| } | Needs, | ||
| cmd = "./build.sh", | ||
| outputs = { | ||
| fq = { glob = "usr/bin/fq" } | OutputBin, | ||
| }, | ||
| # Empty by design: built CGO_ENABLED=0, so the binary is pure-Go static with | ||
| # no DT_NEEDED at all. Same shape as `helm`. If CGO ever creeps back in, the | ||
| # `missing runtime_deps` checker is what notices. | ||
| runtime_deps = [], | ||
| attrs = | ||
| { | ||
| upstream_version = version, | ||
| source_provenance = { category = 'GithubRepo, owner = "wader", repo = "fq" }, | ||
| license_spdx = "MIT", | ||
| } | Attrs, | ||
| tests = { | ||
| smoketest = standaloneTest "/bin/fq --version", | ||
|
|
||
| # DECODE SOMETHING. `fq --version` and even `fq -n '1+1'` pass on a build | ||
| # with every format decoder broken or absent: the jq engine is one Go | ||
| # package and the ~120 format decoders are another, and only the decoders | ||
| # are the reason to ship this tool. | ||
| # | ||
| # The gzip case is the sharp one — asserting on `compression_method` | ||
| # alone would only prove fq read two header bytes. Asserting on | ||
| # `uncompressed` proves it ran the DEFLATE decoder over the payload and | ||
| # got the original bytes back, which is the actual capability. | ||
| decodes = | ||
| { | ||
| class = 'Standalone, | ||
| test_deps = [base], | ||
| cmds = [ | ||
| [ | ||
| "/bin/bash", | ||
| "-c", | ||
| m%" | ||
| set -eu | ||
| msg="hello world hello world" | ||
| printf '%s' "$msg" | gzip -n > /build/t.gz | ||
|
|
||
| # 1. The header field decodes symbolically, not as a raw number. | ||
| m=$(fq -d gzip -r '.members[0].compression_method | tovalue' /build/t.gz) | ||
| [ "$m" = "deflate" ] || { echo "compression_method=$m" >&2; exit 1; } | ||
|
|
||
| # 2. THE REAL ONE: fq inflated the payload and got our bytes back. | ||
| got=$(fq -d gzip -r '.members[0].uncompressed | tostring' /build/t.gz) | ||
| [ "$got" = "$msg" ] || { | ||
| echo "inflate mismatch: '$got' != '$msg'" >&2; exit 1; } | ||
| "% | ||
| ], | ||
| [ | ||
| "/bin/bash", | ||
| "-c", | ||
| m%" | ||
| set -eu | ||
| # 3. A second, unrelated decoder over a real file: fq's own ELF. | ||
| # | ||
| # Derived from uname rather than hardcoded — an earlier package | ||
| # in this loadout pinned an architecture literal and only failed | ||
| # once it reached amd64 CI. fq renders EM_ARM64 as "arm64" and | ||
| # EM_X86_64 as "x86_64" (format/elf/elf.go machineNames). | ||
| case "$(uname -m)" in | ||
| aarch64 | arm64) want=arm64 ;; | ||
| x86_64) want=x86_64 ;; | ||
| *) echo "unsupported arch $(uname -m)" >&2; exit 1 ;; | ||
| esac | ||
| got=$(fq -d elf -r '.header.machine | tovalue' /usr/bin/fq) | ||
| [ "$got" = "$want" ] || { | ||
| echo "elf machine: got '$got', want '$want'" >&2; exit 1; } | ||
| "% | ||
| ], | ||
| # 4. The jq engine itself still works. Last, because on its own it | ||
| # proves the least. | ||
| ["/bin/bash", "-c", "test \"$(fq -n '1+1')\" = 2"], | ||
| ], | ||
| } | Test, | ||
| }, | ||
| } | BuildSpec |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| #!/bin/sh | ||
| set -eu | ||
| export GOROOT=/usr/go | ||
|
|
||
| # CGO_ENABLED=0 is upstream's own build setting (see fq's Makefile) and is what | ||
| # makes the result a pure-Go static binary with no DT_NEEDED — hence the empty | ||
| # runtime_deps in build.ncl. Flipping this back on would silently add a libc | ||
| # dependency that nothing here declares. | ||
| export CGO_ENABLED=0 | ||
|
|
||
| # Reproducibility (see AGENTS.md): -trimpath strips the build directory out of | ||
| # recorded paths and -buildid= clears the non-deterministic build ID. -s -w | ||
| # drop the symbol and DWARF tables, matching the other Go packages here. | ||
| # | ||
| # No -X version stamping: fq keeps its version as a plain const in fq.go, so | ||
| # `fq --version` already reports 0.17.0 from the source tree. A -ldflags -X | ||
| # aimed at a const would be silently ignored — the linker only rewrites vars. | ||
| go build -trimpath -ldflags "-buildid= -s -w" -o fq . | ||
|
|
||
| mkdir -p "$OUTPUT_DIR/usr/bin" | ||
| install -m 755 fq "$OUTPUT_DIR/usr/bin/fq" |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Remove
glibcfrombuild_deps.Line 34 already declares
glibcinruntime_deps. Do not duplicate runtime dependencies inbuild_deps.As per coding guidelines, runtime dependencies do not need to be duplicated in
build_deps.🤖 Prompt for AI Agents
Source: Coding guidelines