Floating point serialization issues in JSON fields #376
Description
We've recently run into some issues serializing specific floating point numbers. We have hit these before, and our understanding is that they stem from Ruby using a different floating point serialization algorithm from Spanner (which spanner runs some API-layer checks to validate).
Prior to this, we were able to work around them by monkeypatching Float#to_s to use a matching grisu2 implementation from [2].
Unfortunately after v2.11.0 of ruby/json[1], this workaround no longer works. ruby/json no longer delegates the serialization back to Float#to_s and instead uses it's own C implementation of grisu2. It's worth noting this isn't a new issue, but has become much more challenging to monkeypatch away now that it doesn't call back to the Ruby layer.
Our understanding is Spanner also uses grisu2 internally, however there seems to be some inconsistencies with implementations in [2] and [1]. In an ideal scenario, the SDK with should maintain a consistent serialization interface with the validations performed by the API layer.
Steps to reproduce
- Add the following to
acceptance/cases/type/json_test.rb:
def test_float_serialization
record = TestTypeModel.create! details: { test: 41.021725 }
record.reload
assert_equal 41.021725, record.details[:test]
end
- Run the test against a remote spanner instance (it passes on the emulator).
- The test should
errorwith with:
1) Error:
ActiveRecord::Type::DateTest#test_float_serialization:
ActiveRecord::StatementInvalid: Google::Cloud::FailedPreconditionError: 9:Invalid value for column details in table test_types: Expected JSON.. debug_error_string:{UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:4005:80e::200a%5D:443 {grpc_message:"Invalid value for column details in table test_types: Expected JSON.", grpc_status:9}}
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1813:in `rescue in commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-v1-1.11.0/lib/google/cloud/spanner/v1/spanner/client.rb:1775:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/google-cloud-spanner-2.28.0/lib/google/cloud/spanner/service.rb:557:in `commit'
lib/spanner_client_ext.rb:29:in `commit_transaction'
lib/activerecord_spanner_adapter/transaction.rb:105:in `commit'
lib/activerecord_spanner_adapter/connection.rb:286:in `commit_transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:305:in `block in commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract_adapter.rb:1142:in `log'
lib/active_record/connection_adapters/spanner_adapter.rb:327:in `log'
lib/active_record/connection_adapters/spanner/database_statements.rb:304:in `commit_db_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:400:in `commit'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:514:in `block in commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:503:in `commit_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:565:in `ensure in block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:564:in `block in within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activesupport-7.1.5.2/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/transaction.rb:532:in `within_new_transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/connection_adapters/abstract/database_statements.rb:344:in `transaction'
lib/active_record/connection_adapters/spanner/database_statements.rb:239:in `transaction'
/Users/jarredhawkins/.rbenv/versions/3.3.0/lib/ruby/gems/3.3.0/gems/activerecord-7.1.5.2/lib/active_record/transactions.rb:212:in `transaction'
lib/activerecord_spanner_adapter/base.rb:28:in `create!'
acceptance/cases/type/json_test.rb:42:in `test_float_serialization'
The problematic value here is 41.021725 -- passing that in as a root node to the JSON (which is valid JSON) also fails.