• uZumaKi Benchmarking Suite
• Index
• Goals
• Current Benchmarks
• Comparison between ZK Circuit Development Frameworks
  • High-Level Language/DSL
    • STARKs
    • SNARKs
  • Low-Level Language
    • STARKs
  • Proof Systems
  • Groth16 Frameworks
  • Plonk Frameworks
  • Arithmetic Operations
  • Elliptic Curve Operations
  • DSL circuit Benchmarks without proving system
  • Assessments
  • Testing Systems Under Heavy Load
  • Type of Platform Specification for Testing
  • Future Work
  • Benchmarks
• Resources
  • Github Repositories
  • Articles
  • Benchmarking

1. Variety of Systems: Numerous ZK proof systems exist, each with unique features and optimizations, necessitating a method for comparison.
2. Targeted Applications: Different systems are tailored for specific use cases, making benchmarking essential to identify the best fit for a particular application.
3. No Universal Solution: Each system has its own trade-offs in terms of efficiency and capabilities, highlighting the need for a comparative analysis.
4. Rapid Evolution: The fast-paced innovation in ZK proofs requires a way to track and evaluate new advancements and their impact.
5. Computational Efficiency: Benchmarking provides insights into the proof generation and verification speeds of different systems.
6. Scalability Insights: Understanding how systems handle increasing computational complexities is crucial for large-scale applications.
7. Security Comparison: Evaluating varying security levels across different systems is vital for ensuring data integrity and privacy.
8. Resource Demand Analysis: Assessing CPU and memory usage helps in determining the practical deployment of ZK systems.
9. Informed Decision-Making: Empirical data from benchmarking aids developers and researchers in choosing the most suitable system.
10. Standardization Aid: Benchmarking supports the standardization process in ZK proofs by establishing performance and best practice benchmarks.

Benchmarking ZK proofs is not just about performance metrics; it's a comprehensive process that evaluates suitability for specific applications, scalability, security, and practicality, thereby guiding the evolving landscape of cryptographic privacy and data integrity solutions.

System Info :
-------------
CPU : Intel i7 12th gen
RAM : 16 GB
GPU : NVIDIA 4050 6 GB, Intel Graphics Driver 6 GB

• Groth16
• Plonk
• Marlin/Marlin'
• Stark

• Gnark
• Rapidsnark
• Arkworks
• Snarkjs
• Bellman
• Zokrates
• Libsnark

• Plonky2
• Halo2
• Aztec (Implementation of Plonk)
• Hercules (Rust-based with Plonk support)

• inv
• mul
• sub
• exp
• add

• g1-scalar-multiplication
• g2-multi-scalar-multiplication
• pairing
• g2-scalar-multiplication
• g1-multi-scalar-multiplication

• Independent of proving scheme limitations: Some proving systems may have limitations or optimizations that can skew the understanding of a DSL's capabilities. Comparing DSLs independently allows for an evaluation that is not influenced by such factors.
• By comparing DSLs independently of specific proving systems, you can focus on the efficiency and optimization of circuit design. This allows for an assessment of how well each DSL facilitates the creation of efficient and optimized circuits.
• Language features, learning curve
• Analysis under heavy load
• Tooling and ecosystem support

• Prover performance
• Verifier performance
• Proof size
• Proof Generation Time (including witness generation time)
• Peak Memory usage during proof generation
• Average CPU Utilization % during proof generation (Reflects parallelization degree)
• Proof cost (Dependent on field and curve efficiency, proof techniques, and computation model)
• EVM Verifier
• External libraries support
• Ease of Use: Learning curve and user-friendliness of each DSL
• Security Features: Built-in security measures of each DSL
• Community and Ecosystem: Community size, resources, documentation, and support
• Version Tracking: Include version numbers of DSLs for updates and improvements
• Parallelization and Scalability: Support for parallel computations and scaling

• Complexity addition via advanced constraints (hashing algorithms, arrays, booleans, data structures, recursion)

• Linux Server: 20 Cores @ 2.3 GHz, 384GB memory
• Macbook M1 Pro: 10 Cores @ 3.2Ghz, 16GB memory
• Icicle: (TBD)

• Benchmarking sequencers
• Benchmarking different zkVMs (e.g., Scroll, Polygon zkEVM, Consensys zkEVM, zkSync, Risc Zero, zkWasm)
• Benchmarking IR compiler frameworks (e.g., zkLLVM)

Miden Starks Fibonacci Prover:

Miden Starks Fibonacci Verify:

Map of curves/Fields with Frameworks and languages

Model of UI for circuit benchmarks: (https://hackmd.io/_uploads/rkrlJP2B6.png)

Metric can be time, ram and proof

Arithmetic Backends

Elliptic curve benchmarks

TBD: Quantitive costs

• Compilation time
• Prover time
• Prover space
• Verifier time
• Verifier space
• Proof size
• Size of public keys and parameters
• Rounds of interaction
• Security level (statistical or computational)

Qualitative costs

• Hardness assumptions (comp vs. PQ)
• Setup assumptions (SRS, URS; universal / specific; updateable?)
• Zero-knowledge (statistical vs comp.)
• Simplicity and ease of verifying correctness
• Parallelization and acceleration
• Can we parallelize or distribute the prover’s computation? Qualitative Costs

• https://github.com/delendum-xyz/zk-benchmarking
• https://github.com/zkCollective/zk-Harness
• https://github.com/celer-network/zk-benchmark
• https://github.com/polybase/zk-benchmarks
• https://github.com/delendum-xyz/zk-benchmarking
• https://github.com/ingonyama-zk/icicle
• C++ CPU Groth16 Prover: https://github.com/MinaProtocol/snark-challenge-prover-reference
• Cuda GPU Groth16 Prover: https://github.com/MinaProtocol/gpu-groth16-prover-3x
• Prize's MSM Implementation: https://github.com/z-prize/test-msm-gpu
• TalDerie Master Research: https://github.com/TalDerei/Masters-Research
• Plonk: Permutations over Lagrange-bases for ecumenical Noninteractive Arguments of Knowledge: https://eprint.iacr.org/2019/953
• Barretenberg https://github.com/AztecProtocol/barretenberg
• Ignition-Verification https://github.com/AztecProtocol/ignition-verification

• https://ethresear.ch/t/benchmarking-zkp-development-frameworks-the-pantheon-of-zkp/14943
• https://hackmd.io/@heliax/SJU01u5fs
• https://eprint.iacr.org/2023/1503
• Aztec's ZK-ZK-Rollup, Looking Behind the Cryptocurtain: https://medium.com/aztec-protocol/aztecs-zk-zk-rollup-looking-behind-the-crypte curtain-2b8af1fca619
• Aleo's Prize Competition: https://www.zprize.io/prizes/accelerating-msm-operations-on-gpu-fpga

• https://www.zk-bench.org/
• https://zkbench.dev/