Skip to content

Optimize S3 Calls #76

@devinmatte

Description

@devinmatte

s3 charges $0.005 per 1000 requests, right now we make 1 request, per stop, per half hour

We should see if there are ways we can optimize the calls slightly without reducing the frequency

https://github.com/transitmatters/gobble/blob/main/src/s3_upload.py#L52-L56

We now upload all files from the current day, every upload time. We should maybe limit to files updated in the last 1 hour again? Would help reduce uploads for commuter rail, weird bus stops and rapid transit at times of shutdowns

These would help reduce upload time as well as cost

Metadata

Metadata

Assignees

No one assigned

    Labels

    optimizationOptimizing some exisiting functionality

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions