Skip to content

🎉 Add a new charts-based data API - #2629

Merged
larsyencken merged 3 commits into
masterfrom
catalog-charts
May 17, 2024
Merged

larsyencken merged 3 commits into
masterfrom
catalog-charts

Conversation

@larsyencken

@larsyencken larsyencken commented May 8, 2024 •

Copy link
Copy Markdown
Contributor

Add a new draft charts-based API for data fetching.

Motivation

We would like to offer simple code snippets that give you the data powering a chart, but we want to prototype what that could look like before committing to baking data in a special way or making a special worker.

The most likely outcome, if we like this code, is that we will later migrate it to a CloudFlare worker and serve the API from there.

A Python charts-based API

from owid.catalog import charts

# get all available charts
slugs = charts.list_charts()

# get a nice data frame, with extra metadata in `df.attrs['metadata']`
df = charts.get_data('life-expectancy')

Technical approach

  • For list_charts()
  • For get_data()
    • Scrape the JSON config from the chart page
    • Fetch data and metadata for every indicator from the indicator-based data API
    • Throw a LicenseError if the data is not redistributable
    • Make a merged data frame
    • Resolve yearIsDay
    • Stuff indicator metadata into df.attrs['metadata']
    • For single-indicator charts, use chart slug as value column name
  • Call a bunch of stuff we want to replace internal, and prefix it all with underscore (e.g. _Indicator) to demonstrate that people shouldn't rely on it

Before merge

Post merge

  • Cut a new owid-catalog release

@owidbot

owidbot commented May 8, 2024 •

Copy link
Copy Markdown
Contributor

Quick links (staging server):

Site Admin Wizard

Login: ssh owid@staging-site-catalog-charts

Chart diff: No new or modified charts. Details

Edited: 2024-05-10 18:20:09 UTC
Execution time: 1.59 seconds

@larsyencken
larsyencken force-pushed the catalog-charts branch 3 times, most recently from d08aba9 to 5f6a135 Compare May 10, 2024 13:58
@larsyencken
larsyencken requested a review from Marigold May 10, 2024 13:59
@larsyencken
larsyencken marked this pull request as ready for review May 10, 2024 13:59

@Marigold Marigold left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice! I just had a couple of minor suggestions. In general, it's very user-friendly.

Comment thread lib/catalog/README.md
Comment thread lib/catalog/README.md Outdated
Comment thread lib/catalog/README.md
Comment thread lib/catalog/owid/catalog/internal.py Outdated
Adds a draft API for fetching the data for a chart.

```
from owid.catalog.charts import Chart

df = Chart('life-expectancy').get_data()
```
- `get_data()` now accepts a URL
- The URL use case is documented first
- Error messages are more helpful
- The docs for the data science API shows how to make it reproducible
@larsyencken
larsyencken merged commit e7d6018 into master May 17, 2024
@larsyencken
larsyencken deleted the catalog-charts branch May 17, 2024 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants