Skip to content

tmahmood/f1-data-pipeline

Repository files navigation

Setting up the project

With Docker

Docker files are provided in docker folder. Start the containers using.

docker compose up -d

the API server will be available at http://localhost:4455

Manually

After cloning/updating the project, provided you are using a virtualenv already, you need to run

# setup base project
pip install -e .

# setup fastf1
cd Fast-F1

pip install -e .

# back to base project
cd ..

This will install required packages and set up the executables.

Setting up the database

Download PostgresSQL: https://www.postgresql.org/download/ Install the appropriate version After installing PostgresSQL, it should be running on port 5432

Connecting the program with the database server

You'll have to create a file named runtime/pgurl and Add the following line

postgresql://postgres:PASSWORD@localhost:PORT/postgres

Replace PASSWORD and PORT with the password, and port that you have set for the database

Migration

To create the necessary tables in the database, run, anytime, after database changes, you'll have to run it again

alembic upgrade head

This will create the required tables

Now you should be able to run the program from the base folder

API server

API server is run by the following command, the ENV variable required for avoiding error with the database folder On Windows, you may not need it

$env:WATCHFILES_IGNORE_PERMISSION_DENIED=0; fastapi dev src/server/fa_server.py --host "localhost"

This will start serving at url http://localhost/8000,

Running the program

Fetch all the data from 2025.

# Manual:
historic_data -y 2025

# Docker:
docker exec -it f1-server historic_data -y 2025

In both cases downloaded data will be located in:

runtime/data/pending/raw/year-2025/session_info

Incomplete events

If there is any event that was incomplete, the script will ignore that event. You will have to run the script later to update the completed event

NOTE: Docker might make the data folder unreadable, due to permissions, in Linux.

Generating telemetry data

To generate telemetry data, you need to run

# Manual:
gen_data -y 2025

# Docker:
docker exec -it f1-server gen_data -y 2025

This will generate all telemetry data required. The process will take some time, depending on your pc configuration. If you want to rerun the data generation, you can run

# Manual:
gen_data -y 2025 -f

# Docker:
docker exec -it f1-server gen_data -y 2025 -f

Live data tracking

Right now, the system is dependent on forked Fast-F1 code to authenticate Start the listener server by running

# Manual:
follow_race -y 2025

# Docker:
docker exec -it f1-server follow_race -y 2025

API server

Please check API Doc for more information about API endpoints

Tests

Comprehensive tests are added to make sure the program is working as expected. Tests can be run using

pytest

This uses sqlite databases for each test, so the original database is unaffected.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages