Skip to content

SQLite: multiline DDL silently drops generated column expressions and collations from schema dump #58131

Description

@PSKK-Git

Steps to reproduce

SQLite stores CREATE TABLE DDL in sqlite_master verbatim. When a table is created from raw SQL written across multiple lines (common with a heredoc in a migration), Rails' SQLite schema introspection silently drops generated column expressions and column collations.

Unlike #58123 (which raises NoMethodError), this fails silently and corrupts db/schema.rb.

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"
  gem "rails"
  gem "sqlite3"
end

require "active_record/railtie"
require "minitest/autorun"
require "stringio"

ENV["DATABASE_URL"] = "sqlite3::memory:"

class TestApp < Rails::Application
  config.load_defaults Rails::VERSION::STRING.to_f
  config.eager_load = false
  config.logger = Logger.new(StringIO.new)
  config.secret_key_base = "secret_key_base"
end
Rails.application.initialize!

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")

class BugTest < ActiveSupport::TestCase
  # Both tables below are semantically identical to their single-line
  # equivalents, which Rails introspects correctly. Only the formatting differs.

  def test_generated_column_expression_survives_multiline_ddl
    connection = ActiveRecord::Base.connection

    connection.execute(<<~SQL)
      CREATE TABLE widgets (
        "id" INTEGER PRIMARY KEY AUTOINCREMENT,
        "price" integer,
        "qty" integer,
        "total" integer GENERATED ALWAYS AS (
          "price" * "qty"
        ) STORED
      )
    SQL

    column = connection.columns("widgets").find { |c| c.name == "total" }

    assert column.virtual?, "column is correctly detected as generated"

    # FAILS: the defining expression is silently dropped.
    assert_equal %{"price" * "qty"}, column.default_function
  end

  def test_collation_survives_multiline_ddl
    connection = ActiveRecord::Base.connection

    connection.execute(<<~SQL)
      CREATE TABLE people (
        "id" INTEGER PRIMARY KEY AUTOINCREMENT,
        "a" varchar COLLATE "NOCASE",
        "b" varchar COLLATE "RTRIM"
      )
    SQL

    columns = connection.columns("people").index_by(&:name)

    assert_equal "NOCASE", columns["a"].collation   # first collated column is fine
    assert_equal "RTRIM",  columns["b"].collation   # FAILS: silently nil
  end

  def test_schema_dump_does_not_silently_corrupt_schema
    connection = ActiveRecord::Base.connection

    connection.execute(<<~SQL)
      CREATE TABLE gadgets (
        "id" INTEGER PRIMARY KEY AUTOINCREMENT,
        "price" integer,
        "qty" integer,
        "total" integer GENERATED ALWAYS AS (
          "price" * "qty"
        ) STORED
      )
    SQL

    output = StringIO.new
    ActiveRecord::SchemaDumper.dump(ActiveRecord::Base.connection_pool, output)
    dump = output.string

    # FAILS: dump contains `as: nil`, which reloads as a permanently-NULL column.
    refute_includes dump, "as: nil"
  end
end

All three tests fail on 8.1.3 and main.

Expected behavior

Multiline DDL should introspect identically to the equivalent single-line DDL:

total  default_function = "\"price\" * \"qty\""
a      collation = "NOCASE"
b      collation = "RTRIM"

Actual behavior

total  default_function = nil      # expression lost
a      collation = "NOCASE"        # first collated column survives
b      collation = nil             # every subsequent one is lost

SchemaDumper then emits a corrupted db/schema.rb:

create_table "gadgets", force: :cascade do |t|
  t.integer "price"
  t.integer "qty"
  t.virtual "total", type: :integer, as: nil, stored: true   # <-- expression gone
end

create_table "people", force: :cascade do |t|
  t.string "a", collation: "NOCASE"
  t.string "b"                                                # <-- COLLATE RTRIM gone
end

Critically, this does not raise on reload. t.virtual "total", as: nil loads happily and produces:

"total" integer GENERATED ALWAYS AS (NULL) STORED

i.e. a column that is permanently NULL for every row, in every environment built from schema.rb, with no error or warning at any point. The lost COLLATE silently changes comparison/sorting semantics (and the behavior of any unique index on that column).

Root cause

Two separate defects in activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb, both triggered by newlines.

1. Column splitting fails on indented DDL (table_structure_sql, line 856):

.split(/,(?=\s(?:CONSTRAINT|"(?:#{Regexp.union(column_names).source})"))/i)

The lookahead allows exactly one whitespace character between the comma and the next column name. Indented multiline DDL has ,\n "price" — a newline plus indentation — so the split never fires and the entire table body is returned as a single chunk.

table_structure_with_collation then does REGEX =~ column_string once per chunk, recording only $1/$2. With one chunk for the whole table, only one column's collation/expression can ever be captured — which is exactly why "a" keeps its collation and "b" loses it.

Confirmed at runtime — table_structure_sql for the multiline table returns:

["\"id\" INTEGER PRIMARY KEY AUTOINCREMENT,\n  \"price\" integer,\n  \"qty\" integer,\n  \"total\" integer GENERATED ALWAYS AS (\n    \"price\" * \"qty\"\n  ) STORED"]

2. GENERATED_ALWAYS_AS_REGEX lacks the m flag (line 784):

GENERATED_ALWAYS_AS_REGEX = /.*"(\w+)".+GENERATED ALWAYS AS \((.+)\) (?:STORED|VIRTUAL)/i

(.+) cannot span newlines without m, so an expression written across lines never matches — the same root cause as #58123.

Proposed fix

Both are required; neither alone is sufficient.

# 1. allow a newline + indentation between columns
.split(/,(?=\s+(?:CONSTRAINT|"(?:#{Regexp.union(column_names).source})"))/i)

# 2. allow the expression to span newlines
GENERATED_ALWAYS_AS_REGEX = /.*"(\w+)".+GENERATED ALWAYS AS \((.+)\) (?:STORED|VIRTUAL)/im

I verified the m flag on its own is actively wrong here: with the whole table body in one chunk, greedy (.+) spanning newlines makes a table with two generated columns match only the last one and drop the first:

current  (\s,  no /m): NO MATCH
/m only  (\s,  /m)   : ["label", "\"price\" || '-' || \"qty\""]   # "total" silently lost
both     (\s+, /m)   : ["total", "\"price\" * \"qty\""]
                       ["label", "\"price\" || '-' || \"qty\""]   # correct

With both changes, the extracted expression retains the surrounding whitespace ("\n \"price\" * \"qty\"\n "), so it likely wants a .strip before being stored in dflt_value.

I'm happy to open a PR with these changes plus regression tests covering multiline generated columns and multiple collated columns.

Related

System configuration

Rails version: 8.1.3 (regexes are identical on main @ a321e18)

Ruby version: 4.0.5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions