Philippe Creux

Rails Dashboards that scale – with SQL and dry-struct

Wed, 09 Jul 2025 00:00:00 +0000

While we can piece together Reports and Dashboards using ActiveRecord, it can be more efficient to use SQL to transform data and dry-struct to turn query results into Ruby objects with strong typing.

Why Not ActiveRecord?

ActiveRecord is a great ORM for typical CRUD operations. While it offers a handful of methods to aggregate data such as .group, .sum, and .joins, they are not sufficient for complex reporting queries.

That’s where raw SQL with Common Table Expressions (CTEs) shines. If you’re not familiar with CTEs (Common Table Expressions), they’re like temporary views that let you transform data from one shape into another. Here we transform operational data into a format suitable for reporting.

Building a simple report: Readership per Team

Zipline users can send messages with associated tasks to multiple teams and track readership and task completion. In this first report, we want to display how many people read messages, grouped by team.

Let’s use CTEs to compute:

message_readership: percent of recipients who read each message
message_team_metrics: readership per communication per team
team_readership: average readership per team

The transformation flow looks like this:

flowchart TD
    %% === Tables Group ===
    subgraph Tables
        direction TB
        Teams[teams]
        Messages[messages]
        MessageTeams[message_teams]
    end

    %% === CTEs Group ===
    subgraph CTEs
        direction TB
        MessageReadership[message_readership]
        MessageTeamMetrics[message_team_metrics]
        TeamReadership[team_readership]
    end

    %% === Edges ===
    Messages --> MessageReadership

    MessageTeams --> MessageTeamMetrics
    MessageReadership --> MessageTeamMetrics

    Teams --> TeamReadership
    MessageTeamMetrics --> TeamReadership

    TeamReadership --> Results

    %% === Styling ===
    classDef table fill:#D6F5D6,stroke:#2C662D,color:#1E4D2B,stroke-width:2px;
    classDef cte fill:#DDEEFF,stroke:#225588,color:#113355,stroke-width:1px,stroke-dasharray: 5;

    class Messages,Teams,MessageTeams table;
    class MessageReadership,MessageTeamMetrics,TeamReadership cte;

Here is the SQL query with CTEs:

WITH message_readership AS (
  SELECT
    m.id AS message_id,
    COUNT(r.*) FILTER (WHERE r.read_at IS NOT NULL) * 1.0 / COUNT(r.*) AS readership
  FROM messages m
  JOIN recipients r ON r.message_id = m.id
  GROUP BY m.id
),

message_team_metrics AS (
  SELECT
    mt.message_id,
    mt.team_id,
    mr.readership
  FROM message_teams mt
  JOIN message_readership mr ON mr.message_id = mt.message_id
),

team_readership AS (
  SELECT
    t.id AS team_id,
    t.name as team_name,
    AVG(mt.readership) AS avg_readership
  FROM teams t
  JOIN message_team_metrics mt ON mt.team_id = t.id
  GROUP BY t.id, t.name
)

SELECT * FROM team_readership

The resulting rows will hydrate a collection of DryStructs:

class TeamReadership < Dry::Struct
  attribute :team_id, Types::Integer
  attribute :team_name, Types::String
  attribute :avg_readership, Types::Float
end

rows = ApplicationRecord.connection.exec_query(team_readership_sql)
structs = rows.map { |row| TeamReadership.new(row.symbolize_keys) }

With this approach, we write a readable SQL query that hydrates a collection of strongly typed ruby objects. We can then build the view layer using DryStructs which is way easier than manipulating a collection of obscure hashes returned by a long SQL query.

Full Dashboard with Nested Structs

Here is a more complex example where we want a dashboard showing metrics at three levels:

Overall metrics (e.g. total messages, avg readership)
Per-message metrics (readership, execution)
Per-team metrics (aggregated per team)

We’re going to implement this as one SQL query with CTEs to perform transformations in two stages:

Staging CTEs to calculate metrics per message x per team
Dashboard CTEs where we compute overal, per-message, and per-team metrics.

We then use Postgres json functions to return these as a json document.

flowchart TD
    %% === Tables Group ===
    subgraph Tables
        direction TB
        Teams[Teams]
        Messages[Messages]
        Tasks[Tasks]
    end

    %% === CTEs Group ===
    subgraph CTEs: Staging
        direction TB
        MessageReadership[message_readership]
        MessageTeams[message_team_metrics]
        MessageTasks[message_tasks]
    end

    subgraph CTEs: Dashboard Sections
        direction TB
        MetricsPerMessage[metrics_per_message]
        OverallMetrics[overall_metrics]
        MetricsPerTeam[metrics_per_team]
    end

    Dashboard["Dashboard (JSON)"]

    %% === Edges ===
    Messages --> MessageReadership
    Messages --> MessageTeams
    Teams --> MessageTeams
    MessageTasks --> MessageTeams

    Messages --> MessageTasks
    Tasks --> MessageTasks

    %%Messages --> MetricsPerMessage
    MessageReadership --> MetricsPerMessage

    %%Teams --> MetricsPerTeam
    MessageTeams --> MetricsPerTeam

    MessageReadership --> OverallMetrics
    MessageTeams --> OverallMetrics

    OverallMetrics --> Dashboard
    MetricsPerMessage --> Dashboard
    MetricsPerTeam --> Dashboard

    %% === Styling ===
    classDef table fill:#D6F5D6,stroke:#2C662D,color:#1E4D2B,stroke-width:2px;
    classDef cte fill:#DDEEFF,stroke:#225588,color:#113355,stroke-width:1px,stroke-dasharray: 5;

    class Messages,Teams,Tasks table;
    class MessageReadership,MessageTeams,MessageTasks,MetricsPerMessage,OverallMetrics,MetricsPerTeam,Dashboard cte;

WITH message_readership AS (
  SELECT
    m.id AS message_id,
    COUNT(r.*) FILTER (WHERE r.read_at IS NOT NULL) * 1.0 / COUNT(r.*) AS readership
  FROM messages m
  JOIN recipients r ON r.message_id = m.id
  GROUP BY m.id
),

message_tasks AS (
  SELECT
    t.message_id,
    COUNT(*) FILTER (WHERE t.completed) * 1.0 / COUNT(*) AS execution
  FROM tasks t
  GROUP BY t.message_id
),

message_team_metrics AS (
  SELECT
    mt.message_id,
    mt.team_id,
    mr.readership,
    mtasks.execution
  FROM message_teams mt
  JOIN message_readership mr ON mr.message_id = mt.message_id
  LEFT JOIN message_tasks mtasks ON mtasks.message_id = mt.message_id
),

overall_metrics AS (
  SELECT
    COUNT(DISTINCT message_id) AS total_messages,
    AVG(readership) AS avg_readership,
    AVG(execution) AS avg_execution
  FROM message_team_metrics
),

metrics_per_message AS (
  SELECT
    m.id AS message_id,
    m.subject,
    AVG(mt.readership) AS readership,
    AVG(mt.execution) AS execution
  FROM messages m
  JOIN message_team_metrics mt ON mt.message_id = m.id
  GROUP BY m.id, m.subject
),

metrics_per_team AS (
  SELECT
    t.id AS team_id,
    t.name,
    AVG(mt.readership) AS readership,
    AVG(mt.execution) AS execution
  FROM teams t
  JOIN message_team_metrics mt ON mt.team_id = t.id
  GROUP BY t.id, t.name
),

dashboard AS (
  SELECT
    row_to_json(om.*) AS overall_metrics,
    json_agg(DISTINCT row_to_json(mm.*)) AS metrics_per_message,
    json_agg(DISTINCT row_to_json(tm.*)) AS metrics_per_team
  FROM overall_metrics om, metrics_per_message mm, metrics_per_team tm
)

SELECT * FROM dashboard;

Here are the corresponding DryStructs:

class MessageMetric < Dry::Struct
  attribute :message_id, Types::Integer
  attribute :subject, Types::String
  attribute :readership, Types::Float
  attribute :execution, Types::Float
end

class TeamMetric < Dry::Struct
  attribute :team_id, Types::Integer
  attribute :name, Types::String
  attribute :readership, Types::Float
  attribute :execution, Types::Float
end

class OverallMetric < Dry::Struct
  attribute :total_messages, Types::Integer
  attribute :avg_readership, Types::Float
  attribute :avg_execution, Types::Float
end

class Dashboard < Dry::Struct
  attribute :overall_metrics, OverallMetric
  attribute :metrics_per_message, Types::Array.of(MessageMetric)
  attribute :metrics_per_team, Types::Array.of(TeamMetric)
end

Let’s hydrate these Structs with the JSON document returned by Postgres:

row = ApplicationRecord.connection.exec_query(dashboard_sql).first
parsed = row.transform_values { JSON.parse(_1) }
dashboard = Dashboard.new(parsed.transform_keys(&:to_sym))

We now have a fully structured dashboard in Ruby objects with validated types that we can use in the view layer.

p
  | Readership:
  = number_to_percentage(@dashboard.overall_metrics.avg_readership)

= render "metrics_per_message", metrics: @dashboard.metrics_per_message
= render "metrics_per_team", metrics: @dashboard.metrics_per_team

Reuse ActiveRecord Scopes

We can easily inject ActiveRecord scopes in our SQL queries so that we don’t duplicate business logic:

sql_scope = Message.last_30_days.for_teams(current_team).to_sql

sql = <<~SQL
  WITH messages AS (
    #{sql_scope}
  ),
  ...
SQL

Performance Tips

Such complex SQL queries can be slow, so we often use EXPLAIN to understand what’s causing poor performance. Here are the strategies I use most of the time:

Filter early: apply date/user filters in the first CTEs.
Duplicate filters across CTEs if needed.
Turn CTEs into temporary tables + indexes.

CREATE TEMP TABLE team_messages AS (
  SELECT ... FROM ...
);

CREATE INDEX ON team_messages (team_id);

WITH message_metrics as (
  SELECT ... FROM team_messages ...
)
...
SELECT * FROM ...

SQL also runs against Data warehouses

Using raw SQL unlocks the possibility of leveraging a data warehouse (Redshift, BigQuery, SnowFlake) to generate reports and dashboards from petabytes of data in seconds.

DryStruct doesn’t care about the data source

One of the best parts of using DryStruct for dashboards is that it decouples the data shape from the data source.

We can hydrate a DryStruct from:

A live SQL query hitting the database or a data warehouse
A hand-crafted YAML or JSON blob for testing
A Faker-powered generator for the demo environment

This is incredibly powerful.

It means we can design and test dashboards without needing access to real data. We can render meaningful samples in staging, ship working demos to sales, and then swap in a real query when we go to production.

It’s also great for writing unit tests — we don’t need to mock a database, we just instantiate structs with test values.

Whether the data comes from petabytes in BigQuery or ten lines of Faker::Lorem, the rest of the app stays the same.

Wrap-Up

You can build high-performance, structured dashboards in Rails using:

SQL with CTEs
ActiveRecord scopes (to avoid duplicating filters)
DryStruct (for type safety and nested hydration)
JSON blobs (to transport nested metrics)

… and nothing prevents you from fetching data from a Data Warehouse or to inject fake data into your dashboards and reports!

Questions? Comments? 👉 pcreux@ruby.social.

Building a Robust Data Synchronization Framework with Rails

Mon, 07 Oct 2024 00:00:00 +0000

At Zipline, we offer a one-stop shop application for retail employees that handles everything from clocking to task management and team communication. To achieve this, we need to integrate with numerous third-party applications with unique requirements. We decided to develop a satellite service called ZipIO for data synchronization. In this post, I’ll review our journey of building this robust and scalable system.

Key Design Principles

Language and Framework: We chose Ruby on Rails for its simplicity and productivity. However, we structured our core logic as POROs (Plain Old Ruby Object) to decouple the core logic from the engine that runs it.
Modular Architecture: We broke down the workflow into discrete steps: Signals, Commands, Outcomes, and Tasks. This modular approach allows for easy reuse of logic across different third-party integrations.
Asynchronous Processing: While we support synchronous operations for specific use cases, 99% of our workflows run asynchronously for optimal performance and scalability.
Robust Error Handling: Each step in the workflow is designed to handle errors gracefully, ensuring smooth operation even when issues arise.
Full Visibility: We prioritized comprehensive logging and monitoring to make troubleshooting and maintenance as painless as possible.

Defining workflows

Steps

Our workflow consists of four main kind of steps:

Signals: These are triggers, often from API calls or webhooks, that initiate a workflow.
Commands: Actions performed in response to signals, such as synchronizing data with another app.
Outcomes: The results of commands, which can be success, failure, or skipped.
Tasks: Scheduled operations that poll third-party services for new data to sync. They create Signals, in turn.

Each step in the workflow is designed as a standalone Ruby object that takes a data input, performs an action, and returns the next step(s) to be executed. Since each step defines the downstream step(s), it can perform complex routing depending on input data or user configuration. For example, a Signal could return several Commands to sync an event to multiple third-party services.

flowchart LR

A[Signal: New Order]
B[Command: Sync to service A]
C[Command: Sync to service B]
BB[Outcome: Success]
CB[Outcome: Success]

A --> B
A --> C
B --> BB
C --> CB

The data inputs are defined as Dry::Struct. Each step declares the type(s) of step it returns, adding an extra layer of type safety and allowing us to generate documentation automatically.

Here is an example:

# This signal is created when we receive a new order via a webhook.
class Stripe::Signals::NewOrder < ZipIO::Step
  data_struct [Stripe::Structs::Order]

  emits MyStore::Commands::SaveOrder

  # Convert Stripe Order to a Generic Struct and return a Command to create the order
  def call
    order = Transformer.call(data, to: Generic::Structs::Order)

    MyStore::Commands::CreateOrder.new(order)
  end
end

class Stripe::Commands::CreateOrder < ZipIO::Step
  # Create an order via an API and return an Outcome according to the response.
 def call
   response = api_client.orders.post(data)

   if response.success?
     MyStore::Outcome::OrderCreated.new(response.parsed_body)
   else
     MyStore::Outcome::OrderCreateFailure.new(response.parsed_body)
  end

  private

  def api_client
    # ...
  end
end

# Two outcomes. They are the final steps of the workflow so they don't do anything.
Stripe::Commands::OrderCreated = Class.new(ZipIO::Outcomes::Success)
Stripe::Commands::OrderCreateFailure = Class.new(ZipIO::Outcomes::Failure)

# Order received from Stripe.
class Stripe::Structs::Order < DryStruct
  attribute :amount, Types::Decimal
  attribute :order_id, Types::String
  attribute :user_id, Types::String
  attribute :email, Types::String
  attribute :payment_type, Types::String
end

# Generic order used throughout the system.
class Generic::Structs::Order < DryStruct
  attribute :amount, Types::Decimal
  attribute :order_id, Types::String
  attribute :integration_key, Types::String
  attribute :email, Types::String
  attribute :payment_source, Types::String.enum('credit_card', 'debit_card')
end

Transforming data

The core logic is to transform data from one schema (ex: Stripe Order) to another (ex: Generic Order). We created a simple library inspired by csv-importer to define data transformation using a DSL.

class StripeOrderToGenericOrder < ZipIO::Transformer
  # map `amount` to `amount`
  attribute :amount

  # map `stripe_order_id` to `order_id`
  attribute :order_id, source: :stripe_order_id

  # build a custom integration key
  attribute :integration_key, source: ->(data) {
    [data.fetch(:user_id), data.fetch(:order_id)].join("-")
  }

  # downcase email
  attribute :email, transform: -> { _1.&downcase }

  # Define mapping
  attribute :payment_source, source: :payment_type, transform: Mapper.new(
    "cc" => "credit_card",
    "dc" => "debit_card",
    # ...
  )
end

Using a proper naming convention, we can look up the transformers on the fly:

Transformer.call(stripe_order, to: Generic::Structs::Order)
# => <# Generic::Structs::Order ...>

Live documentation

Since Steps, Structs, and Transformers are defined using DSLs, we can introspect them to generate documentation. This documentation is available in our Admin UI so that non-technical folks (Product Owners, Account Managers, …) understand how the data flows and how it’s transformed.

We traverse the graph of emit-ed steps to render a flow chart:

flowchart LR

A[Stripe::Signals::Webhook] --> B[Stripe::Signals::PaymentSucceeded]

A[Stripe::Signals::Webhook] --> C[Stripe::Signals::NewOrder]

A[Stripe::Signals::Webhook] --> D[Stripe::Outcome::IgnoredWebhookEvent]

C --> E[MyStore::Commands::CreateOrder]

E --> F[MyStore::Outcome::OrderCreated]

E --> G[MyStore::Outcome::OrderCreateFailure]

L[Shopify::Signals::Webhook] --> M

M[Shopify::Signals::NewOrder] --> E

We look up all Structs and Transformers to display them in a human-friendly way. Here is for example the documentation for StripeOrderToGenericOrder:

Source Target Transform

amount ➡️ amount

stripe_order_id ➡️ order_id

Proc

[data.fetch(:user_id), data.fetch(:order_id)].join("-")

➡️

integration_key

➡️

Proc

-> { _1.&downcase }

payment_source

➡️

payment_type

"cc"	➡️	"credit_card"
"dc"	➡️	"debit_card"

So far, we’ve defined the steps, structs, and transformations. Let’s see how we make data flow through them.

Processing a Workflow

Processing a Workflow synchronously is straightforward.

# Run a workflow synchronously and return a collection of child steps.
# Ex: [ Stripe::Signals::NewOrder, [ MyStore::Commands::CreateOrder, [ MyStore::Outcome::OrderCreated ] ] ]
def run(step)
  return step if step.is_a?(Symbol) # e.g. :success, :skip, ...

  next_steps = Array(step.call)

  next_steps.map do |next_step|
    run(next_step)
  end
end

We use this approach to perform integration tests where we want to exerce an entire workflow. On a production environment, we persist Pipelines and Steps to the database and we process them asynchronously.

Database Structure

We use a minimal set of Rails models:

Pipeline: A specific instance of a pipeline
PipelineStep: Individual steps within a pipeline

Asynchronous Processing

When a signal is received:

A new pipeline is created and persisted in the database.
A job is enqueued to process the first step.
Each step, when processed, can trigger subsequent steps, which are then enqueued as separate jobs.

This approach allows for excellent scalability, fault tolerance, and visibility.

Error Handling and Retry Mechanism

By persisting each step in the database, we can retry failed steps without restarting the entire workflow. This is particularly useful for long sequences of API calls where only one step might fail.

Scalability

Our service runs on Heroku, using Judoscale to auto-scale background workers. This setup allows us to handle peak loads efficiently by spinning up additional workers as needed.

Example

In the example below, we handle Stripe webhooks. Each webhook can contain multiple events, so we start a pipeline where the first step breaks down each event into its own Signal.

class StripeWebhooksController < ApplicationController
  # POST /webhooks/stripe
  # The webhooks can contain multiple events.
  # Build a Webhook signal from the raw payload and start a Pipeline.
  def create
    signal = Stripe::Signals::Webhook.new(params)
    Pipeline.start!(signal)
  end
end

class Stripe::Signals::Webhook < ZipIO::Signal
  # Return an array of individual signals.
  # These signals will be processed in parallel.
  def call(webhook)
    webhook.events.filter_map do |event|
      case event.type
      when 'payment_intent.succeeded'
        Stripe::Signals::PaymentSucceeded.new(event)
      when 'orders.succeeded'
        # See definition above.
        Stripe::Signals::NewOrder.new(event)
      else
        Stripe::Outcome::IgnoredWebhookEvent.new(event)
      end
    end
  end
end

At a high level, the Pipeline implementation looks like this:

class Pipeline < ApplicationRecord
  # Persist the signal as the first step of the pipeline and process it asynchronously.
  def self.start!(signal)
    PipelineStep.enqueue(signal)
  end
end

class PipelineStep < ApplicationRecord
  # Persist the step and process it asynchronously.
  def self.enqueue(step, pipeline: nil)
    pipeline ||= Pipeline.create!

    step_model = create!(data: step.attributes, step_class: step.class.name, pipeline: pipeline)

    step_model.process_async!
  end

  # Enqueue a job that calls `process!`
  def process_async!
    ProcessPipelineStepJob.perform_async(self)
  end

  # Constantize the persisted step, trigger `call`, and enqueue the resulting steps.
  def process!
    step = step_class.constantize.new(data) # ex: Stripe::Signals::Webhook
    # Trigger the step
    resulting_step = step.call

    return if resulting_step.is_a?(Symbol) # :success, :fail, :skip

    Array(resulting_step).each do |step|
      PipelineStep.enqueue(step, pipeline: self.pipeline)
    end
  end
end

Instrumentation, visibility, and monitoring

Since every single step is persisted in the database, writing an admin interface that gives complete visibility into each pipeline and its steps is straightforward.

Thanks to this admin interface, most debugging sessions are painless. There’s no need to dig into the logs or open a Rails console; everything is nicely laid out in a web UI.

Every step run is sent to Datadog to help us monitor the system’s health and alert us when failure rates are high.

Conclusion

I’m very happy with how ZipIO turned out. We succeeded to built a framework that allows us to define the core logic by writing simple and elegant code. After three years of operation, the codebase defines 125 steps and 50 transformers. We have processed 1 billion steps, and our service continues to be a pleasure to work with and expand.

Questions? Comments? 👉 pcreux@ruby.social.

Organize your Rails codebase with aaa engines

Thu, 26 Sep 2024 00:00:00 +0000

Once a Rails app has 50+ models and controllers, it’s time to organize the code by feature using Rails engines.

We give each feature a directory containing the app code, tests, migrations, and routes specific to it. It looks like a mini-rails app that only contains the code related to that feature.

Working with these mini-rails apps is a breeze! You can fully expand the app/* directories, and they all neatly fit in the file explorer on the left-hand-side of your screen. It’s so satisfying to navigate such a small set of files.

I helped extract such engines on two large apps, and they were enthusiastically adopted. It doesn’t require much effort as long as you move files around without attempting to isolate the engine from the rest of the app.

Alright, how do we go about it?

`aaa` engines

At Zipline, we decided to name the directory that contains these engines aaa. In a file browser, it looks like this:

- zipline-app/
  - aaa/
    - admin/
    - checkout/
    - forum/
      - app/controllers/forum/
        - posts_controller.rb
        - comments_controller.rb
      - app/models/forum/
        - post.rb
        - comment.rb
      - config/routes.rb
      - db/migrate/20240802_forum_posts.rb
      - lib/
      - test/
      forum.gemspec
    - surveys/
  - app/
    - controllers/
    - models/
      - 100+ models :cry:
  - test/

Why aaa? So that it sits at the top of the file browser, above the app directory. 😅

I worked on an application where that directory was called engines. While it makes more sense than aaa, having it in the middle of the core application directories was messy—especially when you expanded a couple of core and engine directories. Now, while aaa doesn’t stand for anything, you could argue that it’s an acronym for “Auxiliary Application Addons”. 🙃 “aaa engines” is now part of Zipline’s common language. It also helps us differentiate them from other kinds of engines (e.g. engines offering capabilities such as view components, form auto-save, etc).

Engine boilerplate code

We leverage Rails Engines to make Rails aware of app code, routes, and migrations stored in an aaa subdirectory. We are NOT building configurable engines, isolated from the rest of the app, that get mounted at a specific path, like devise or active admin.

These simple aaa engines only need three files to be picked-up by rails: a gemspec, a lib/XXX.rb, and an engine.rb file. Here is the boilerplate code for an engine called forum:

# aaa/forum/forum.gemspec
$:.push File.expand_path("lib", __dir__)

Gem::Specification.new do |spec|
  spec.name        = "forum"
  spec.version     = "1.0.0"
  spec.authors     = ["AAA"]
  spec.summary     = "The Forum AAA Engine"
end

# aaa/forum/lib/forum.rb
require "forum/engine"

module Forum
  def self.table_name_prefix
    "forum_"
  end
end

# aaa/forum/lib/forum/engine.rb
module Forum
  class Engine < ::Rails::Engine
  end
end

Let’s make rails aware of the aaa engine by adding a line to the Gemfile:

# Gemfile
gem 'forum', path: 'aaa/forum'

Copy all the things!

Now that your engine is set up, you can move existing code from the core directories to the aaa engine directory.

mkdir -p aaa/forum/app/models
mv app/models/forum aaa/forum/app/models/

Here is a script to make it a bit easier: extract-forum-to-aaa-engine.rb

Restart your rails app and it will just work - including code reloading in the development environment. ✌️

Going further

Namespacing

Models, controllers, and views don’t get automagically prefixed. So a model called Forum::Post must be defined under aaa/forum/app/models/forum/post.rb. This also allows you to define classes in multiple namespaces (or not namespaced).

# aaa/forum/app/models/forum/post.rb
class Forum::Post < ApplicationRecord
end

Adding migrations

With the following lines added to your engine.rb, you can store migrations under aaa/forum/db/migrations.

module Forum
  class Engine < ::Rails::Engine
    initializer :append_migrations do |app|
      append_engine_paths("db/migrate", app)
    end

    private

    def append_engine_paths(type, app)
      return if app.root.to_s.match root.to_s

      config.paths[type].expanded.each do |expanded_path|
        app.config.paths[type] << expanded_path
      end
    end
  end
end

To generate a migration, you must run rails g migration ... at the root level and then copy the migration from db/migrations to aaa/forum/db/migrations.

Adding routes

Routes defined in aaa/forum/config/routes.rb will be loaded.

Rails.application.routes.draw do
  namespace :forum do
    resources :posts
    resources :comments
  end
end

Tests

You run tests from the root directory.

$> bin/rails test aaa/forum/test

You should update your CI config to run tests from aaa engines alongside the core rails app tests.

Third-party gems and services

If you use i18n-tasks, you should register your engine paths to i18n-config.yml.

If you use Sentry, I recommend adding stack.abs_path:**/aaa/** +app to “STACK TRACE RULES” so that the stack trace from aaa engines is considered part of the app and displayed by default.

That’s all, folks!

It is that simple. We leverage Rails Engines to make Rails aware of the code that lives under aaa sub-folders, and we enjoy working with smaller sets of files.

Some may want to isolate aaa engines from the rest of the application by adding a public API. In my experience, that adds quite a bit of weight and complexity but can be a good fit once you reach Shopify’s scale (see Shopify’s Monolith - they use Rails engines to encapsulate components and Packwerk to enforce isolation).

aaa engines make it easier to experiment with new patterns or libraries (ViewComponent, event sourcing, etc) as the scope of the experiment can be limited to a specific aaa engine.

I put together a sample rails app with an aaa engine called Forum. Have a look on GitHub!

Give it a try, and let me know what you think!

Thanks to Rémi Mercier his feedback and comments.

Handling webhooks in Ruby on Rails

Fri, 19 Jan 2024 00:00:00 +0000

I had to handle webhooks on most web applications that I worked on: email events (deliveries, bounces, etc), payment/subscription notifications, inbound emails, etc. They can all contain large payloads and multiple events, and they expect the app to return promptly a successful response otherwise they eventually retry.

I recommend to:

Make the controller as simple as possible and process webhooks asynchronously
Process individual events contained in the webhook asynchronously
Persist the payload to the database to improve visibility for monitoring and debugging purposes

1. Make the controller as simple as possible and process webhooks asynchronously

I’ve seen apps where the webhook is entirely handled in the controller. That can work for small payloads and simple logic but it doesn’t scale™. Production level payload size, data, and logic are likely to make your app take several seconds to process the webhook – that will block a web process and the service sending the webhook could time out as they tend to have short request timeouts (~5 to 10 seconds).

You also don’t control the schema of the webhook and while it should remain fairly stable nothing prevents the third-party service from updating it. For example , I ran into timestamp format changes multiple times.

So I recommend processing the webhook asynchronously and making the controller as simple as possible.

2. Process individual events contained in the webhook asynchronously

Webhooks often contain a batch of events. To prevent a failure to affect all events, I tend to make the controller parse the payload to enqueue one job per event. Your job will be responsible for processing just one event - nice!

3. Persist the payload to the database to improve visibility for monitoring and debugging purposes

When the logic or the payload are complex you want to have good visibility into the webhooks and to be able to re-process them.

I find persisting the raw payloads in an ActiveRecord model the best way to support this.

The controller persists the webhook payload (or webhooks events) in ActiveRecord models A job processes the webhook record The job marks the webhook record as completed or failed.

Here is a sample code:

class StripeWebhook < ApplicationModel
  # id, type, payload:json, created_at, updated_at, status, details:json

  enum { pending: 10, completed: 20, failed: 30, skipped: 40 }

  after_create :enqueue_job

  def failed!(error)
    update!(status: :failed, details: { error_message: error.message, error_type: error.class.name, error_backtrace: error.backtrace })
  end

  def enqueue_job
    StripeWebhookJob.constantize.perform_async(self)
  end
end

class StripeWebhookJob < ApplicationJob
  attr_reader :webhook

  def perform(webhook)
    @webhook = webhook

    ApplicationRecord.transaction do
      webhook.with_lock do
        break unless webhook.pending?

        execute(webhook)

      rescue => e
        webhook.failed!(e)
        raise e
      end
    end
  end

  def execute(webhook)
    # real code here
  end
end

You could extract the “BackgroundJob model and job” behaviors into concerns if this is something you want to apply to other webhooks or background jobs (ex: generating a report, importing a CSV, etc).

Oh, and it’s also worth mentioning that webhooks are not always sent, received, and processed in order – so watch out for race conditions, especially when it has to do with subscriptions or payments. :)

Questions? Comments? 👉 pcreux@ruby.social.

Snapshot testing with dbt

Sun, 05 Nov 2023 00:00:00 +0000

I really enjoy building data warehouses using dbt.

dbt is a simple framework to define data transformation using plain SQL. We use it to transform data from our operational databases into tables for our datawarehouse.

Our dbt project can target both Postgres and BigQuery. This enables us to build the datawarehouse locally on Postgres. While we can quickly verify that the SQL syntax is valid, testing the correctness of the resulting dataset is a manual step.

dbt comes with a minimal testing framework to detect missing or duplicate entries but I find it insufficient make changes to the code with confidence. I didn’t have much success finding alternatives to dbt built-in tests so I decided to write unit tests. It didn’t take long to realize that writing unit tests to validate 100+ models would be quite tedious. Moreover, in my experience, the most common errors are due to changes in unrelated models so I needed something to help me quickly see that something was off without much effort.

I decided to give a try to using snapshot tests and it’s been of a great help so far!

Snapshot tests involve capturing the output and manually verifying it. It’s often used to detect unexpected user interface changes. Here, the output is the collection of tables created by dbt and their data.

We store the data in yaml files and check it in our GitHub repository. This allows us to verify visually that the expected changes are happening and spot any regression.

Here is a screenshot from a pull request where we’re making a change to capitalize the string device_type and fallback to "Unknown" instead of leaving it blank.

model/analytics.events.sql is a dbt model
test/input/app/event_stream_events.yml is a yaml dump from the operational database
test/output/analytics/events.yml is a yaml dump from the datawarehouse model

How does it work?

Our dbt models can target both Postgres and BigQuery so that we can build the data warehouse locally on Postgres.

The data warehouse is built from a handful of application databases. On production, these are synced over to BigQuery.

When we perform a full test run, the following happens:

we load the database SQL schema as individual Postgres schemas
we load input data from yaml files into these schemas
we run dbt locally
we dump the resulting (output) schemas in yaml files

Here is an example:

test/
  schemas/
    app_1_db.sql
    app_2_db.sql
  input/
    app_1_db/
      table_1.yml
      table_2.yml
    app_2_db/
      table_1.yml
  output/
    dataset_1/
      model_1.yml
    dataset_2/
      model_1.yml

test/schemas

These are simple postgres dumps from the source databases. We regularly update them as the application databases get changed. We follow the convention where the name of the file is the name of the postgres schema.

input/SCHEMA/TABLE.yml

These yaml files contain the data for the given schema and table. Here is an example:

# input/app_1_db/users.yml
john:
  id: 1
  name: John
  birthdate: 2000-01-01
  created_at: 2018-01-01
sam:
  id: 2
  name: Sam
  birthdate: 2000-01-08
  created_at: 2018-01-01

These files were manually exported from a demo database and curated to limit their size. Each table has anywhere between 1 to 10 records at the moment. These files are maintained by hand.

In order to keep the output stable, we define all attributes including ids. We also make the ids unique across tables to catch bad joins (ex: join users on users.id = team_id).

output/DATASET/MODEL.yml

These yaml files show the data of the tables created in the data warehouse in yaml files. Example:

# output/dw/users.yml
- id: user/1
  name: John
  birth_month: 1
  birth_day: 8
  created_at: 2018-01-01
- id: user/2
  name: Sam
  birth_month: 1
  birth_day: 1
  created_at: 2018-01-01

The records are sorted by id to keep the output files stable.

We are a ruby shop, so the tool is built in ruby and executed using rake (the ruby equivalent of make, ant, yarn, etc).

We keep the build & snapshot step snappy by skipping steps that don’t need to re-run. So we only load schema if the schema files have changed, we only load data if the data files have changed, and we only run dbt if the dbt files have changed (this is made possible because we materialize models as SQL views).

Fast feedback loop and increased confidence

When we make changes to the code, we always run rake instead of running dbt run so that we can check the snapshot diffs (git diff). Reviewing a diff is much quicker and easier than checking the result in a DB Client and it surfaces unintended consequences in other tables that we wouldn’t have seen otherwise. This fast feedback loop makes development much more enjoyable and we can push up pull requests with high confidence in the change we’re making.

Reviewing pull requests is also a breeze as we can review the snapshot alongside the code to validate the behavior change.

Now, this can only be possible by ensuring the input files contain a wide variety of combinations. When a change is not reflected in the output, we update the input files to make sure that the behavior is surfaced in the output files. When we run into a bug on production, the fix includes updates to the input files acting as a regression test.

The time spent writing and maintaining this small snapshot testing framework paid off in a couple of weeks as it sped up both our development and review process, and it lowered the amount of bug shipped to production.

We’re unlikely to release this tool as an open source project because it is writen in Ruby (dbt is Python!) and it is quite specific to our needs. I’m happy to answer any questions on Mastodon.

Event Sourcing made Simple

Sun, 01 Apr 2018 00:00:00 +0000

Five Practices for Robust Ruby on Rails applications

Mon, 01 Feb 2016 00:00:00 +0000

Be nice to others and your future-self: use Data Objects

Wed, 01 Jul 2015 00:00:00 +0000

Weather forecast retrospective meeting

Wed, 01 Jul 2015 00:00:00 +0000

Elegant tests with Truth Tables

Wed, 01 Apr 2015 00:00:00 +0000

Healthy Codebase and Preparatory Refactoring

Sat, 01 Nov 2014 00:00:00 +0000

Gourmet Service objects

Tue, 01 Apr 2014 00:00:00 +0000