How to structure a bot for long run

For large bots, processing large data sets, or scraping many pages, we recommend following these guidelines to produce robust automations.

# Use Google Sheets for large runs

We recommend using Google Sheets as your primary data source in nearly all cases.

This applies even if you have files like CSVs (they can be imported into Sheets), and even if you are sending data from Zapier or Make.

Please avoid reading and writing to a single sheet for your whole automation - this makes your automations harder to structure.

Instead, we recommend using separate "INPUT" and "OUTPUT" sheets, with the following (example) structure.

# Structure a bot for large runs

The advantage of this design pattern is that both "Write data to a Google Sheet" and "Delete rows from Google Sheet" operations are performed inside the loop. This means that for every iteration of the loop, new data is saved, and the processed row is deleted. If the bot stops, it can be restarted and will pick up from where it left off.

Design pattern: Long running bots
  • 1 Read data from a Google Sheet
  • 2 Loop through data
    • 2.1 Go to page
    • 2.2 Insert steps here
    • 2.3 Write data to a Google Sheet
    • 2.4 Delete row from Google Sheet

# Set a maximum runtime

Axiom.ai contains a setting for maximum runtime. This is useful if you have a large bot and don't want it to accidentally use up all of the runtime for your plan.

You can alter this setting from the configuration page for your individual axiom.

For the cloud, the maximum runtime cannot exceed the single run limit for your plan.

# Send data from Zapier, Make & other platforms

When sending data from Zapier or Make, it's possible to either send data directly, or to write data to Google Sheets first.

Sending data directly works fine for small quantities of data.

However, if your data-set contains many columns, or more than 1 row, we recommend using Google Sheets to transfer data:

How
  • First, structure your bot to read from a Google Sheet
  • Secondly, write data to Google Sheets from Zapier / Make
  • Finally, trigger bots from Zapier / Make. Sending data is not required, as your bot will use the Google Sheet instead
Prev: How to login
Next: How to use logic in your automation