ONNX Runtime bindings for Crystal
-
Install ONNX Runtime
Download and install the ONNX Runtime from the official releases.
For Linux:
# Example for Linux wget https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-linux-x64-1.21.0.tgz tar -xzf onnxruntime-linux-x64-1.21.0.tgz export ONNXRUNTIME_DIR=/path/to/onnxruntime-linux-x64-1.21.0
For macOS:
# Example for macOS (arm64) curl -L https://github.com/microsoft/onnxruntime/releases/download/v1.21.0/onnxruntime-osx-arm64-1.21.0.tgz -o onnxruntime-osx-arm64-1.21.0.tgz tar -xzf onnxruntime-osx-arm64-1.21.0.tgz export ONNXRUNTIME_DIR=/path/to/onnxruntime-osx-arm64-1.21.0
-
Add the dependency to your
shard.yml:dependencies: onnxruntime: github: kojix2/onnxruntime.cr
-
Run
shards install
require "onnxruntime"
# Load a model
session = OnnxRuntime::InferenceSession.new("path/to/model.onnx")
# Print model inputs and outputs
puts "Inputs:"
session.inputs.each do |input|
puts " #{input.name}: #{input.type} #{input.shape}"
end
puts "Outputs:"
session.outputs.each do |output|
puts " #{output.name}: #{output.type} #{output.shape}"
end
# Prepare input data
input_data = {
"input_name" => [1.0_f32, 2.0_f32, 3.0_f32]
}
# Run inference
result = session.run(input_data)
# Process results
result.each do |name, data|
puts "#{name}: #{data}"
end
# Explicitly release resources
session.release_session
OnnxRuntime::InferenceSession.release_envDownload the MNIST model: mnist-12.onnx (raw
require "onnxruntime"
# Load the MNIST model
session = OnnxRuntime::InferenceSession.new("mnist-12.onnx")
# Create a dummy input (28x28 image draw 1)
input_data = Array(Float32).new(28 * 28) { |i| (i % 14 == 0 ? 1.0 : 0.0).to_f32 }
# Run inference
result = session.run({"Input3" => input_data}, ["Plus214_Output_0"], shape: {"Input3" => [1_i64, 1_i64, 28_i64, 28_i64]})
# Get the output probabilities
probabilities = result["Plus214_Output_0"].as(Array(Float32))
# Find the digit with highest probability
predicted_digit = probabilities.index(probabilities.max)
puts "Predicted digit: #{predicted_digit}"
# Explicitly release resources
session.release_session
OnnxRuntime::InferenceSession.release_envCurrently, you must explicitly release resources when you're done with them:
# Create and use session
session = OnnxRuntime::InferenceSession.new("path/to/model.onnx")
result = session.run(input_data)
# When finished, explicitly release resources
session.release_session
OnnxRuntime::InferenceSession.release_envFor long-running applications like web servers, consider setting up signal handlers to ensure resources are properly released on shutdown:
Signal::INT.trap do
puts "Shutting down..."
session.release_session
OnnxRuntime::InferenceSession.release_env
exit
endSee the examples directory for more detailed implementations.
Why do you need to manually free memory?
Previously, the following error was displayed on macOS.
libc++abi: terminating due to uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Program received and didn't handle signal ABRT (6)
This seems to be related to multithreading and mutex. According to the AI, this is difficult to solve with finalize, so we tried to solve it by creating a reference counter, but we were unable to solve it in the end. If you can solve this problem, please create a pull request!
The code is generated by AI and may not be perfect. Please feel free to contribute and improve it.
- Fork it (https://github.com/kojix2/onnxruntime.cr/fork)
- Create your feature branch (
git checkout -b my-new-feature) - Commit your changes (
git commit -am 'Add some feature') - Push to the branch (
git push origin my-new-feature) - Create a new Pull Request