One of our biggest projects through COVID-19 has been our global map of aid consisting of over 10 thousand charitable organizations, nonprofits, NGOs providing access to critical resources (oxygen cylinders, ICU beds, ventilators, etc.) for thousands of people in need worldwide.

We're currently storing all the data on Algolia but it's heavily limited in ways you can easily update/add data, as well as add new attributes to our data (better location mapping, tagging to highlight certain markers during specific crises, etc.).

As such, we're currently trying to sync all the records from Algolia to Airtable (you've likely heard of it in CVT) to make it a bit more generalizable and adaptable for future crises.

In the colab here, we've done most of the fetching and transforming of the data from Algolia to a generic JSON format (as well as general cleanup). Now, we need to do some parsing and Google Maps API calls to populate some extra map and location data and then we can migrate all our data over to Algolia.

Changelog of the data model:

TO DO:

Add:

• loc.description_clean ([MAP] Extract location data using the Google Maps API from each marker #1726)
• loc.country ([MAP] Extract location data using the Google Maps API from each marker #1726)
• loc.city ([MAP] Extract location data using the Google Maps API from each marker #1726)
• airtableID (unique identifier to the same record in airtable)
• icon (similar to https://mapahelp.me/)
• tags (easier to highlight certain markers during certain events)

Update:

• contact.phone (use phonenumbers package to parse and properly format using international standards)

DONE:

Add:

• source (name of source, original source object renamed to sourceInfo)

Merge:

• contact.general, contact.getHelp, contact.volunteers -> contact

Rename:

• type.type -> type
• type.services -> services
• source -> sourceInfo

Remove:

• _geoLoc (_geoloc is the correct one)
• loc.latlng (since _geoloc contains the same info)
• airtable
• _verifiedAt
• _ft... (firetable metadata)
• id (algolia uses objectID which contains the same info)

Algolia <-> Airtable Master Data Model Mapping