Skip to content

Add manual OAS generation benchmark - #138

Draft
numbata wants to merge 2 commits into
mainfrom
benchmark/generation-time
Draft

numbata wants to merge 2 commits into
mainfrom
benchmark/generation-time

Conversation

@numbata

@numbata numbata commented Sep 8, 2026

Copy link
Copy Markdown
Owner

Problem

grape-oas profiles memory use during OAS generation, but it has no repeatable way to measure generation time as an API's route count grows. This makes performance work harder to evaluate across OAS dialects.

Fix

Add a manually invoked benchmark:generation Rake task inspired by Grape's compile_many_routes.rb. It builds synthetic APIs with 100, 500, and 1,000 routes, warms up each exporter, verifies the generated path count, and reports median generation time for OAS 2.0, 3.0, and 3.1.

Route counts and iteration count can be overridden with environment variables. The benchmark uses Ruby's monotonic clock, so it adds no dependency. It is not connected to CI or release tasks.

Example

bundle exec rake benchmark:generation
ROUTE_COUNTS=50,250 ITERATIONS=10 bundle exec rake benchmark:generation

Output is tab-separated for easy comparison:

routes  dialect  median_ms
100     oas2     6.79
100     oas3     6.72
100     oas31    6.82

Schema before / after

No change to emitted output. This PR adds a standalone benchmark that calls the existing GrapeOAS.generate API; library generation code is unchanged.

Backward compatibility

  • Public API surface: unchanged.
  • Emitted OAS shape: unchanged.
  • New runtime/dev dependencies: none.

🤖

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Memory Profile Report

Summary

Metric Value
Iterations 5
API Endpoints 18
Avg Allocated 1.78 MB
Avg Retained 888.00 B
Retained Trend 0.0%
Memory Stable Yes
Potential Leak No

Iteration Details

# Allocated Retained Objects
1 1.78 MB 888.00 B 24954
2 1.78 MB 888.00 B 24954
3 1.78 MB 888.00 B 24954
4 1.78 MB 888.00 B 24954
5 1.78 MB 888.00 B 24954

Top Memory Allocations (by location)

# Avg Memory Location
1 208.63 KB lib/grape_oas/doc_key_normalizer.rb:11
2 121.21 KB lib/grape_oas/api_model_builders/concerns/oas_utilities.rb:18
3 89.69 KB lib/grape_oas/type_resolvers/primitive_resolver.rb:44
4 82.03 KB lib/grape_oas/api_model/schema.rb:26
5 82.03 KB lib/grape_oas/api_model/schema.rb:32
6 62.03 KB lib/grape_oas/schema_constraints.rb:10
7 59.53 KB lib/grape_oas/exporter/concerns/schema_indexer.rb:63
8 51.80 KB lib/grape_oas/api_model/schema.rb:43
9 48.91 KB vendor/bundle/ruby/4.0.0/gems/securerandom-0.4.1/lib/securerandom.rb:71

Generated by Memory Profile workflow • Commit: bc0d33b

@github-actions

github-actions Bot commented Sep 8, 2026 •

Copy link
Copy Markdown

Danger Report

No issues found.

View run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant