Add Semaphore JMH benchmarks#10712
Conversation
Three benchmark classes covering different scenarios: - SemaphoreContentionBenchmark: contention with varying fibers/permits - SemaphoreFastPathBenchmark: no-contention fast path overhead - SemaphoreMultiPermitBenchmark: multi-permit acquire (withPermits(n)) Each compares ZIO Semaphore vs JDK Semaphore vs Cats Effect Semaphore.
There was a problem hiding this comment.
Pull request overview
Adds a new JMH benchmark suite for Semaphore performance comparisons across ZIO, JDK, and Cats Effect, intended to support ongoing semaphore optimization work with repeatable measurements.
Changes:
- Introduces a contention benchmark with configurable fiber/permit counts.
- Adds a “fast path” (no contention) benchmark to measure acquire/release overhead.
- Adds a multi-permit benchmark to exercise
withPermits(n)/ partial-allocation behavior.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| sem <- ZIO.succeed(new JSemaphore(permits)) | ||
| fiber <- | ||
| ZIO.forkAll(List.fill(fibers)(repeat(opsPerFiber) { | ||
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) |
There was a problem hiding this comment.
JSemaphore.acquire() is a blocking, interruptible call (and may throw InterruptedException). Wrapping it in ZIO.succeed(...) runs the blocking call on the compute pool and turns interruptions/exceptions into defects, which can skew results or fail the benchmark. Prefer ZIO.attemptBlockingInterrupt(sem.acquire()) (or ZIO.blocking(ZIO.attempt(sem.acquire()))) in the acquire effect.
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) | |
| ZIO.acquireReleaseWith(ZIO.attemptBlockingInterrupt(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) |
| unsafeRun(for { | ||
| sem <- ZIO.succeed(new JSemaphore(100)) | ||
| _ <- repeat(ops)( | ||
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) | ||
| ) |
There was a problem hiding this comment.
Same issue as above: JSemaphore.acquire() is blocking/interruptible and shouldn’t be run via ZIO.succeed(...) on the compute pool. Use ZIO.attemptBlockingInterrupt(sem.acquire()) (or equivalent) for the acquire effect to keep blocking off the main executor and preserve interruption semantics.
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire(acquireSize)))(_ => ZIO.succeed(sem.release(acquireSize)))(_ => | ||
| ZIO.succeed(1) | ||
| ) |
There was a problem hiding this comment.
JSemaphore.acquire(acquireSize) is a blocking, interruptible call; wrapping it in ZIO.succeed(...) runs it on the compute pool and turns interruptions/exceptions into defects. Use ZIO.attemptBlockingInterrupt(sem.acquire(acquireSize)) (or ZIO.blocking(ZIO.attempt(...))) for the acquire effect.
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire(acquireSize)))(_ => ZIO.succeed(sem.release(acquireSize)))(_ => | |
| ZIO.succeed(1) | |
| ) | |
| ZIO.acquireReleaseWith(ZIO.attemptBlockingInterrupt(sem.acquire(acquireSize)))(_ => | |
| ZIO.succeed(sem.release(acquireSize)) | |
| )(_ => ZIO.succeed(1)) |
| @OutputTimeUnit(TimeUnit.SECONDS) | ||
| @Warmup(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Measurement(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Fork(3) |
There was a problem hiding this comment.
These benchmarks spawn/manage their own concurrency via fibers; it’s important that JMH doesn’t also add extra OS threads implicitly. Many existing benchmarks pin this explicitly via @Threads(1) (e.g. benchmarks/src/main/scala/zio/ForkAllBenchmark.scala:10). Consider adding @Threads(1) here to keep results stable/reproducible.
| @Fork(3) | |
| @Fork(3) | |
| @Threads(1) |
| @OutputTimeUnit(TimeUnit.SECONDS) | ||
| @Warmup(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Measurement(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Fork(3) |
There was a problem hiding this comment.
Consider adding @Threads(1) to this benchmark class as well (consistent with other single-driver benchmarks like benchmarks/src/main/scala/zio/ForkAllBenchmark.scala:10). Since the benchmark already models contention via fibers, extra JMH threads can distort the intended scenario.
| @Fork(3) | |
| @Fork(3) | |
| @Threads(1) |
| @OutputTimeUnit(TimeUnit.SECONDS) | ||
| @Warmup(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Measurement(iterations = 3, timeUnit = TimeUnit.SECONDS, time = 3) | ||
| @Fork(3) |
There was a problem hiding this comment.
Consider adding @Threads(1) here too, for the same reason as the other semaphore benchmarks: contention is modeled inside the benchmark via fibers, so multiple JMH threads can unintentionally multiply load and skew comparisons.
| @Fork(3) | |
| @Fork(3) | |
| @Threads(1) |
| /** | ||
| * Benchmarks for the fast path (no contention). | ||
| * | ||
| * When permits > fibers, every acquire succeeds on the fast path. This measures | ||
| * the overhead of the semaphore machinery itself. | ||
| */ |
There was a problem hiding this comment.
The comment says this measures fast-path semaphore overhead, but the benchmark currently includes semaphore allocation/construction on every invocation (Semaphore.make(100L) / new JSemaphore(100) / CESemaphore(100L)). Consider moving semaphore creation into a @Setup (e.g., per-iteration/trial) and reusing it so the measurement reflects acquire/release overhead rather than construction cost.
| sem <- Semaphore.make(permits.toLong) | ||
| fiber <- ZIO.forkAll(List.fill(fibers)(repeat(opsPerFiber)(sem.withPermit(ZIO.succeed(1))))) | ||
| _ <- fiber.join | ||
| } yield ()) | ||
|
|
||
| @Benchmark | ||
| def javaSemaphore(): Unit = | ||
| unsafeRun(for { | ||
| sem <- ZIO.succeed(new JSemaphore(permits)) | ||
| fiber <- | ||
| ZIO.forkAll(List.fill(fibers)(repeat(opsPerFiber) { | ||
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) | ||
| })) | ||
| _ <- fiber.join |
There was a problem hiding this comment.
ZIO.forkAll(...) collects results into a Chunk, but the results are discarded. This adds allocation overhead unrelated to semaphore performance. Consider using ZIO.forkAllDiscard(...) here (and similarly in the other methods using forkAll) so the benchmark focuses on semaphore contention rather than result aggregation.
| sem <- Semaphore.make(permits.toLong) | |
| fiber <- ZIO.forkAll(List.fill(fibers)(repeat(opsPerFiber)(sem.withPermit(ZIO.succeed(1))))) | |
| _ <- fiber.join | |
| } yield ()) | |
| @Benchmark | |
| def javaSemaphore(): Unit = | |
| unsafeRun(for { | |
| sem <- ZIO.succeed(new JSemaphore(permits)) | |
| fiber <- | |
| ZIO.forkAll(List.fill(fibers)(repeat(opsPerFiber) { | |
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) | |
| })) | |
| _ <- fiber.join | |
| sem <- Semaphore.make(permits.toLong) | |
| _ <- ZIO.forkAllDiscard(List.fill(fibers)(repeat(opsPerFiber)(sem.withPermit(ZIO.succeed(1))))) | |
| } yield ()) | |
| @Benchmark | |
| def javaSemaphore(): Unit = | |
| unsafeRun(for { | |
| sem <- ZIO.succeed(new JSemaphore(permits)) | |
| _ <- | |
| ZIO.forkAllDiscard(List.fill(fibers)(repeat(opsPerFiber) { | |
| ZIO.acquireReleaseWith(ZIO.succeed(sem.acquire()))(_ => ZIO.succeed(sem.release()))(_ => ZIO.succeed(1)) | |
| })) |
Summary
Adds comprehensive JMH benchmarks for
Semaphore, comparing ZIO vs JDK vs Cats Effect across three scenarios:SemaphoreContentionBenchmark— Contention with varying fiber counts (2, 10, 50, 100) and permit counts (1, 5, 10). The most realistic scenario.SemaphoreFastPathBenchmark— No contention (100 permits, single fiber). Measures the overhead of the semaphore machinery itself on the fast path.SemaphoreMultiPermitBenchmark— Multi-permitwithPermits(n)with varying acquire sizes (1, 3, 5) and 10 total permits. Tests the partial allocation path.Context
These benchmarks support the ongoing Semaphore optimization work:
Semaphorecode #10378 — Allocation optimization (current implementation)Semaphore#9662 — Lock-free rewriteHaving a shared benchmark suite allows objective comparison of different implementation strategies.
Test plan
sbt benchmarks/compilepassessbt "benchmarks/Jmh/run -i 3 -wi 3 -f 3 -t 1 .*Semaphore.*"