football_analytics

A data pipeline that ingests football (soccer) match and event data from the InStat Football API into PostgreSQL and trains a RandomForest expected-goals (xG) model on the event stream.

This repo is the ingestion + xG-model half of a larger football stack. Its companion repo, football_data, is the visualization / analysis layer (pass maps, radars, xG over/under- performance graphs) that reads from the same PostgreSQL database. They are complementary — this one fills the database and computes xG; football_data turns it into charts and analysis.

Status: this is a 2018-era pipeline. It depends on the (paid, closed) InStat feed, so it cannot run end-to-end without InStat credentials and a live endpoint. The code has been cleaned up (bug fixes, dependency hygiene) but no API migration or modernization beyond that has been done.

InStat API  ──>  get_data/      ──>  PostgreSQL  ──>  expected_goals/  ──>  pred_stats table
(api_calls)      (raw stats &        (match,           (build_X +            (xg, xa,
                  events ingest)      event tables)      RandomForest xG)     xg_chain, xg_buildup)

                 update_db/  ──>  incremental updates of existing matches

• get_data/ — pulls games, squads, team stats, player stats and game events from the InStat API and writes them to PostgreSQL. Also loads the static lookup tables in get_data/csvs/ (actions, zones, positions, etc.).
• expected_goals/ — builds the feature matrix from the event stream (build_X.py, get_model_variables.py), trains/loads a RandomForestRegressor via GridSearchCV (xg_model.py), and writes per-shot xG / xA / xG-chain / xG-buildup back to PostgreSQL (get_predictive_stats.py).
• update_db/ — re-runs ingestion + xG for a date range over already-known matches.
• lib_common/ — shared database helpers (db_handle.py, dbconnectors.py).

• Python 3
• A running PostgreSQL database
• InStat Football API access (an API_ID / API_KEY)

1. Install dependencies:

   pip install -r requirements.txt

2. Create the two local credential modules from the provided templates. Both are git-ignored and must never be committed:

   cp api_keys.py.example api_keys.py   # InStat API_ID / API_KEY
   cp db_data.py.example  db_data.py    # DB_HOST / DB_NAME / DB_USERNAME

   The PostgreSQL password is read from your local .pgpass / PGPASSWORD, not from db_data.py.

Each entry point lives in a python/ subfolder and is run as a script from that folder (the modules import sibling files by appending directories to sys.path).

1. Initial load — create tables and ingest a date range. Running main_get_data.py defaults to setup=True, which prompts before rebuilding the database (it runs DROP TABLE ... CASCADE):

   cd get_data/python && python3 main_get_data.py

2. Incremental update — re-ingest a date range and compute xG for those matches:

   cd update_db/python && python3 main.py

3. xG model only — build features, train/load the RandomForest and write predictive stats for the configured date range:

   cd expected_goals/python && python3 main_xg.py

Date ranges and league/season ids are configured in the constants_*.py modules (START_DATE, END_DATE, LEAGUES, SEASONS).

xg_model.py trains a sklearn.ensemble.RandomForestRegressor with a GridSearchCV parameter sweep (constants_xg.MODEL_PARAMS) on a feature matrix derived from the event stream. The trained model is pickled under model/ and reused on subsequent runs. Outputs written to the pred_stats table:

• xg — expected goals for each shot
• xa — expected assists
• pos_xg_chain — xG of the possession a player was involved in
• pos_xg_buildup — xG-chain excluding the shot and the key pass

• The InStat feed is a paid, vendor-specific endpoint served over plain HTTP and is hardcoded in get_data/python/api_calls.py; the pipeline cannot run without valid InStat credentials and a live endpoint.
• lib_common/dbconnectors.py is a generic multi-DB framework; this project only uses its PostgreSqlDb path.