Steps to reproduce
SQLite stores CREATE TABLE DDL in sqlite_master verbatim. When a table is created from raw SQL written across multiple lines (common with a heredoc in a migration), Rails' SQLite schema introspection silently drops generated column expressions and column collations.
Unlike #58123 (which raises NoMethodError), this fails silently and corrupts db/schema.rb.
require "bundler/inline"
gemfile(true) do
source "https://rubygems.org"
gem "rails"
gem "sqlite3"
end
require "active_record/railtie"
require "minitest/autorun"
require "stringio"
ENV["DATABASE_URL"] = "sqlite3::memory:"
class TestApp < Rails::Application
config.load_defaults Rails::VERSION::STRING.to_f
config.eager_load = false
config.logger = Logger.new(StringIO.new)
config.secret_key_base = "secret_key_base"
end
Rails.application.initialize!
ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")
class BugTest < ActiveSupport::TestCase
# Both tables below are semantically identical to their single-line
# equivalents, which Rails introspects correctly. Only the formatting differs.
def test_generated_column_expression_survives_multiline_ddl
connection = ActiveRecord::Base.connection
connection.execute(<<~SQL)
CREATE TABLE widgets (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"price" integer,
"qty" integer,
"total" integer GENERATED ALWAYS AS (
"price" * "qty"
) STORED
)
SQL
column = connection.columns("widgets").find { |c| c.name == "total" }
assert column.virtual?, "column is correctly detected as generated"
# FAILS: the defining expression is silently dropped.
assert_equal %{"price" * "qty"}, column.default_function
end
def test_collation_survives_multiline_ddl
connection = ActiveRecord::Base.connection
connection.execute(<<~SQL)
CREATE TABLE people (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"a" varchar COLLATE "NOCASE",
"b" varchar COLLATE "RTRIM"
)
SQL
columns = connection.columns("people").index_by(&:name)
assert_equal "NOCASE", columns["a"].collation # first collated column is fine
assert_equal "RTRIM", columns["b"].collation # FAILS: silently nil
end
def test_schema_dump_does_not_silently_corrupt_schema
connection = ActiveRecord::Base.connection
connection.execute(<<~SQL)
CREATE TABLE gadgets (
"id" INTEGER PRIMARY KEY AUTOINCREMENT,
"price" integer,
"qty" integer,
"total" integer GENERATED ALWAYS AS (
"price" * "qty"
) STORED
)
SQL
output = StringIO.new
ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection_pool, output)
dump = output.string
# FAILS: dump contains `as: nil`, which reloads as a permanently-NULL column.
refute_includes dump, "as: nil"
end
end
All three tests fail on 8.1.3 and main.
Expected behavior
Multiline DDL should introspect identically to the equivalent single-line DDL:
total default_function = "\"price\" * \"qty\""
a collation = "NOCASE"
b collation = "RTRIM"
Actual behavior
total default_function = nil # expression lost
a collation = "NOCASE" # first collated column survives
b collation = nil # every subsequent one is lost
SchemaDumper then emits a corrupted db/schema.rb:
create_table "gadgets", force: :cascade do |t|
t.integer "price"
t.integer "qty"
t.virtual "total", type: :integer, as: nil, stored: true # <-- expression gone
end
create_table "people", force: :cascade do |t|
t.string "a", collation: "NOCASE"
t.string "b" # <-- COLLATE RTRIM gone
end
Critically, this does not raise on reload. t.virtual "total", as: nil loads happily and produces:
"total" integer GENERATED ALWAYS AS (NULL) STORED
i.e. a column that is permanently NULL for every row, in every environment built from schema.rb, with no error or warning at any point. The lost COLLATE silently changes comparison/sorting semantics (and the behavior of any unique index on that column).
Root cause
Two separate defects in activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb, both triggered by newlines.
1. Column splitting fails on indented DDL (table_structure_sql, line 856):
.split(/,(?=\s(?:CONSTRAINT|"(?:#{Regexp.union(column_names).source})"))/i)
The lookahead allows exactly one whitespace character between the comma and the next column name. Indented multiline DDL has ,\n "price" — a newline plus indentation — so the split never fires and the entire table body is returned as a single chunk.
table_structure_with_collation then does REGEX =~ column_string once per chunk, recording only $1/$2. With one chunk for the whole table, only one column's collation/expression can ever be captured — which is exactly why "a" keeps its collation and "b" loses it.
Confirmed at runtime — table_structure_sql for the multiline table returns:
["\"id\" INTEGER PRIMARY KEY AUTOINCREMENT,\n \"price\" integer,\n \"qty\" integer,\n \"total\" integer GENERATED ALWAYS AS (\n \"price\" * \"qty\"\n ) STORED"]
2. GENERATED_ALWAYS_AS_REGEX lacks the m flag (line 784):
GENERATED_ALWAYS_AS_REGEX = /.*"(\w+)".+GENERATED ALWAYS AS \((.+)\) (?:STORED|VIRTUAL)/i
(.+) cannot span newlines without m, so an expression written across lines never matches — the same root cause as #58123.
Proposed fix
Both are required; neither alone is sufficient.
# 1. allow a newline + indentation between columns
.split(/,(?=\s+(?:CONSTRAINT|"(?:#{Regexp.union(column_names).source})"))/i)
# 2. allow the expression to span newlines
GENERATED_ALWAYS_AS_REGEX = /.*"(\w+)".+GENERATED ALWAYS AS \((.+)\) (?:STORED|VIRTUAL)/im
I verified the m flag on its own is actively wrong here: with the whole table body in one chunk, greedy (.+) spanning newlines makes a table with two generated columns match only the last one and drop the first:
current (\s, no /m): NO MATCH
/m only (\s, /m) : ["label", "\"price\" || '-' || \"qty\""] # "total" silently lost
both (\s+, /m) : ["total", "\"price\" * \"qty\""]
["label", "\"price\" || '-' || \"qty\""] # correct
With both changes, the extracted expression retains the surrounding whitespace ("\n \"price\" * \"qty\"\n "), so it likely wants a .strip before being stored in dflt_value.
I'm happy to open a PR with these changes plus regression tests covering multiline generated columns and multiple collated columns.
Related
System configuration
Rails version: 8.1.3 (regexes are identical on main @ a321e18)
Ruby version: 4.0.5
Steps to reproduce
SQLite stores
CREATE TABLEDDL insqlite_masterverbatim. When a table is created from raw SQL written across multiple lines (common with a heredoc in a migration), Rails' SQLite schema introspection silently drops generated column expressions and column collations.Unlike #58123 (which raises
NoMethodError), this fails silently and corruptsdb/schema.rb.All three tests fail on 8.1.3 and
main.Expected behavior
Multiline DDL should introspect identically to the equivalent single-line DDL:
Actual behavior
SchemaDumperthen emits a corrupteddb/schema.rb:Critically, this does not raise on reload.
t.virtual "total", as: nilloads happily and produces:i.e. a column that is permanently
NULLfor every row, in every environment built fromschema.rb, with no error or warning at any point. The lostCOLLATEsilently changes comparison/sorting semantics (and the behavior of any unique index on that column).Root cause
Two separate defects in
activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb, both triggered by newlines.1. Column splitting fails on indented DDL (
table_structure_sql, line 856):The lookahead allows exactly one whitespace character between the comma and the next column name. Indented multiline DDL has
,\n "price"— a newline plus indentation — so the split never fires and the entire table body is returned as a single chunk.table_structure_with_collationthen doesREGEX =~ column_stringonce per chunk, recording only$1/$2. With one chunk for the whole table, only one column's collation/expression can ever be captured — which is exactly why"a"keeps its collation and"b"loses it.Confirmed at runtime —
table_structure_sqlfor the multiline table returns:2.
GENERATED_ALWAYS_AS_REGEXlacks themflag (line 784):(.+)cannot span newlines withoutm, so an expression written across lines never matches — the same root cause as #58123.Proposed fix
Both are required; neither alone is sufficient.
I verified the
mflag on its own is actively wrong here: with the whole table body in one chunk, greedy(.+)spanning newlines makes a table with two generated columns match only the last one and drop the first:With both changes, the extracted expression retains the surrounding whitespace (
"\n \"price\" * \"qty\"\n "), so it likely wants a.stripbefore being stored indflt_value.I'm happy to open a PR with these changes plus regression tests covering multiline generated columns and multiple collated columns.
Related
mroot cause inVIRTUAL_TABLE_REGEX(raises rather than corrupting)schema.rb#56969, Add support for SQLite3 full-text-search and other virtual tables #52354 — prior fixes to the same DDL-parsing areaSystem configuration
Rails version: 8.1.3 (regexes are identical on
main@ a321e18)Ruby version: 4.0.5