Olaf

Olaf is a small Ruby wrapper for warehouse queries, running them in BigQuery.

olaf depends on nothing by itself. Each driver loads its own client library, and only when it is used, so add to your Gemfile the ones you configure:

If you don't have Olaf, try this:

$ gem install olaf

Olaf helps developers to represent warehouse queries as objects, to have more control in the code and in tests.

class FetchUsers
  include Olaf::QueryDefinition

  template './big_query/users_in_department.sql'

  argument :department_id

  row_object User
end

query = FetchUsers.prepare(department_id: 1337)

Olaf.execute(query)
=> [#<User id: 41, department_id: 1337, name: 'Ian'>]

One driver, which every query runs on:

Olaf.configure(olaf_driver: Olaf::Fake)  # ideal for testing

Or several, which queries pick by name. The ones that don't declare a driver run on the default:, the first one given when it is not specified:

Olaf.configure(
  drivers: {
    big_query: Olaf::BigQuery.new(project: 'carwow', labels: { service: 'flatmin' }),
    fake: Olaf::Fake.new
  },
  default: :big_query
)

class FetchUsers
  include Olaf::QueryDefinition

  driver :big_query

  template './big_query/users_in_department.sql'

  argument :department_id
end

Olaf.instance(:big_query) returns that driver. A single configured driver serves every query, whatever they declare, which is what keeps Olaf::Fake covering all of them in tests:

Olaf.configure(olaf_driver: Olaf::Fake)

Olaf.instance.register_result(FetchUsers, [{ id: 41 }])
Olaf.instance.register_result(FetchUsers, [{ id: 42 }], with: { department_id: 1337 })

Whichever driver runs the query, a failure it reports is wrapped into Olaf::QueryExecutionError, carrying the query metadata. A BigQuery connection that keeps dropping can be the exception: see below.

Olaf::BigQuery.new(
  project: 'carwow',
  credentials: JSON.parse(ENV['BIGQUERY_CREDENTIALS']),  # ambient Google credentials when omitted
  maximum_bytes_billed: 5 * Olaf::BigQuery::GIGABYTE,    # default: 1 GiB
  labels: { service: 'flatmin', country: 'uk' },
  retry_delays: [1, 2]                                   # default: retry twice, after 1s then 2s
)

The query runs as a job, so that it can be capped and labelled:

• Olaf's :placeholders are rewritten to BigQuery's @named parameters, for declared arguments only, and bound as query parameters. Arguments declared as: :literal are substituted into the SQL before that, as usual.

• Every job is capped with maximum_bytes_billed, and labelled with the driver labels plus the name of the template. A query that scans more than the cap raises its own, which keeps a regression failing instead of quietly costing money:

  driver :big_query, maximum_bytes_billed: 10 * Olaf::BigQuery::GIGABYTE

• A call whose connection drops mid-request is retried once after each of the retry_delays. Waiting for the job is retried on its own, so a dropped status check doesn't start a second job. A drop arrives in one of two forms, and both are retried:

  • A bare OpenSSL::SSL::SSLError, which is how google-apis-core 0.x reports an SSL read that ends early. Once the retries run out it is raised as it is, not wrapped, so a job runner such as Sidekiq can retry it in turn.
  • A Google::Cloud::Error caused by a Google::Apis::TransmissionError. google-apis-core 1.x reports every dropped connection this way, and 0.x does the same for a reset connection. Once the retries run out it is wrapped into Olaf::QueryExecutionError, like any other error the API reports.

Olaf.execute returns the rows. Olaf.instance(:big_query).run(query) returns the finished job instead, for its statistics:

job = Olaf.instance(:big_query).run(query)
job.bytes_processed
job.cache_hit?
job.data.all.to_a