Skip to content
This repository was archived by the owner on Jun 11, 2025. It is now read-only.
This repository was archived by the owner on Jun 11, 2025. It is now read-only.

Automatically add Athena partition when new test results are available #50

@binarylogic

Description

@binarylogic

In order for Athena to query new benchmarking data is must know about the partition. This is achieved by running a repair query:

MSCK REPAIR TABLE vector_tests

This query is expensive and takes a few minutes to execute. This is because it's listing every file in the bucket in order to discover new partitions. A much more efficient method is to add the partition directly:

ALTER TABLE vector_tests ADD PARTITION (...) location 's3://test-results.vector.dev/../../../'

For example, this could be done automatically by listening for new S3 files being placed in the bucket and firing a lambda to run this query.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions