ml4t-data

Unified market data acquisition and storage for quantitative research workflows.

This library is one of six interconnected libraries supporting the machine learning for trading workflow described in Machine Learning for Trading:

Together they cover data infrastructure, feature engineering, modeling, signal evaluation, strategy backtesting, and live deployment.

Quantitative research requires consistent, reproducible access to market data from multiple sources. ml4t-data provides:

• DataManager as the unified OHLCV interface for registry providers with that capability
• 20+ provider adapters covering equities, crypto, futures, forex, macro, prediction markets, and factors
• Automated storage in Hive-partitioned Parquet format with metadata tracking
• Incremental updates, gap detection, and backfill via CLI
• Built-in data validation (OHLC invariants, deduplication, anomaly detection)
• Futures module for CME/ICE bulk downloads with continuous contract construction
• COT module for CFTC Commitment of Traders weekly reports
• Resilience: rate limiting, retry with exponential backoff, gap detection

The goal is to support an ongoing research workflow rather than one-off downloads. Data is stored locally, tracked for freshness, and queryable with tools like DuckDB or Polars.

ML4T Data supports stable CPython 3.12 through 3.14 on Linux, macOS, and Windows. Python 3.15 is not supported until the core dependency stack passes the complete compatibility suite on all three operating systems. Releases 0.1.0 and 0.1.1 predate this upper bound, so an unpinned installation on Python 3.15 may select one of those older releases. Use Python 3.12 through 3.14 instead.

pip install ml4t-data

from ml4t.data import DataManager

dm = DataManager()

# Fetch and store
dm.fetch("AAPL", "2020-01-01", "2024-12-31", provider="yahoo")

# Load from local storage
data = dm.load("AAPL", "2020-01-01", "2024-12-31")

# Batch load multiple symbols
prices = dm.batch_load(["AAPL", "MSFT", "GOOGL"], "2020-01-01", "2024-12-31")

# Incremental update
dm.update("AAPL")

# List what's stored
symbols = dm.list_symbols()
metadata = dm.get_metadata("AAPL")

Providers expose capability-specific methods. OHLCV providers use fetch_ohlcv, economic-series providers may also use fetch_series, and factor providers use fetch.

from datetime import UTC, datetime, timedelta

from ml4t.data.providers import YahooFinanceProvider, CoinGeckoProvider, FREDProvider

# Equities
provider = YahooFinanceProvider()
data = provider.fetch_ohlcv("AAPL", "2020-01-01", "2024-12-31")

# Crypto
last_complete_day = datetime.now(UTC).date() - timedelta(days=1)
crypto = CoinGeckoProvider().fetch_ohlcv(
    "bitcoin",
    str(last_complete_day - timedelta(days=6)),
    str(last_complete_day),
)

# Economic data
# Reads FRED_API_KEY from the environment
fred = FREDProvider().fetch_series("GDP", "2020-01-01", "2024-12-31")

CryptoCompare account registration was unavailable during the 0.1.0 release review, so no live contract evidence was obtained. The adapter remains available for evaluation, but CryptoCompare is not part of the release-qualified provider set until its contract passes on a release commit.

Bulk download and continuous contract construction for CME/ICE products:

from ml4t.data.futures import FuturesDownloader, ContinuousContractBuilder

# Bulk download via Databento (parent symbology)
downloader = FuturesDownloader(config)
downloader.download()  # Downloads ES, NQ, CL, GC, etc.

# Build continuous contracts with configurable roll logic
builder = ContinuousContractBuilder()
continuous = builder.build(contracts_df, roll_method="volume")

Book-focused interface with profiling:

from ml4t.data.futures import FuturesDataManager

fm = FuturesDataManager.from_config("config.yaml")
fm.download_all()
data = fm.load_ohlcv("ES")
profile = fm.generate_profile("ES")

CFTC weekly positioning data for futures markets:

import polars as pl

from ml4t.data.cot import COTFetcher, combine_cot_ohlcv, create_cot_features

fetcher = COTFetcher()
official_schedule = pl.read_parquet("cot-release-schedule.parquet")
cot_data = fetcher.fetch_product(
    "ES",
    start_year=2015,
    end_year=2024,
    release_schedule=official_schedule,
)

# Point-in-time combination with OHLCV
combined = combine_cot_ohlcv(ohlcv_data, cot_data)

# Generate weekly features without counting forward-filled daily rows as new reports
features = create_cot_features(combined)

The schedule must contain one report_date and timezone-aware available_at timestamp per report. CFTC publishes a tentative schedule for only the latest 13 months, so retain the exact release timestamps you use for historical research. Shutdown catch-up releases and other exceptions must use their actual publication timestamps. See the official CFTC release schedule.

Simplified interfaces for the ML4T book workflow:

from ml4t.data.etfs import ETFDataManager
from ml4t.data.crypto import CryptoDataManager

# 50 diversified ETFs via Yahoo Finance
etf_dm = ETFDataManager.from_config("config.yaml")
etf_dm.download_all()
aapl = etf_dm.load_ohlcv("AAPL")

# Crypto premium index via Binance Public
crypto_dm = CryptoDataManager.from_config("config.yaml")
crypto_dm.download_premium_index()

# Fetch specific symbols
ml4t-data fetch -s AAPL -s MSFT -s GOOGL --provider yahoo --start 2020-01-01

# Incremental update
ml4t-data update --symbol AAPL

# Validate data quality
ml4t-data validate --symbol AAPL --anomalies

# Check storage status
ml4t-data status --detailed

# List available data
ml4t-data list-data

# Export to CSV/JSON/Excel
ml4t-data export --symbol AAPL --format-type csv --output aapl.csv

# Get symbol info
ml4t-data info --symbol AAPL

Configuration-driven batch updates:

storage:
  base_path: ~/data/market

datasets:
  - name: sp500_daily
    provider: yahoo
    symbols_file: symbols/sp500.txt
    frequency: daily
    start_date: 2015-01-01

  - name: crypto
    provider: coingecko
    symbols: [bitcoin, ethereum, solana]
    frequency: daily
    initial_load_days: 29

Data is stored in Hive-partitioned Parquet:

~/data/market/
├── yahoo/daily/symbol=AAPL/data.parquet
├── yahoo/daily/symbol=MSFT/data.parquet
└── coingecko/daily/symbol=bitcoin/data.parquet

Query with DuckDB or Polars:

import duckdb

result = duckdb.execute("""
    SELECT * FROM read_parquet('~/data/market/yahoo/daily/**/*.parquet')
    WHERE symbol IN ('AAPL', 'MSFT')
    AND date >= '2024-01-01'
""").pl()

from ml4t.data.validation import OHLCVValidator, ValidationReport

validator = OHLCVValidator()
report = validator.validate(data)
# Checks: high >= low, high >= open/close, low <= open/close
# Detects: duplicates, gaps, anomalies

Anomaly detection:

from ml4t.data.anomaly import AnomalyManager, ReturnOutlierDetector, VolumeSpikeDetector

manager = AnomalyManager([
    ReturnOutlierDetector(),
    VolumeSpikeDetector(),
])
report = manager.detect(data)

• Getting Started - quick start guide
• Configuration - YAML config reference
• Storage - Hive partitioning and backends
• Incremental Updates - update strategies and gap detection
• Data Quality - validation and anomaly detection
• CLI Reference - command-line interface
• Provider Selection Guide - choosing providers
• Creating a Provider - extending with new sources

• Polars-based: Native Polars DataFrames throughout
• Capability-specific schemas: OHLCV, economic-series, factor, and specialized providers expose contracts suited to their data
• Async support: OHLCV providers implementing the async protocol can use parallel batch operations
• Metadata tracking: Last update timestamps, row counts, date ranges
• Resilience: Rate limiting, retry with exponential backoff, gap detection
• Storage layouts: Partitioned Hive and single-file Parquet storage
• Type-safe: Full type annotations throughout

• ml4t-engineer: Feature engineering and technical indicators
• ml4t-diagnostic: Signal evaluation and statistical validation
• ml4t-backtest: Event-driven backtesting
• ml4t-live: Live trading with broker integration

git clone https://github.com/ml4t/data.git ml4t-data
cd ml4t-data
uv sync
uv run pytest tests/ -q
uv run ty check

MIT License - see LICENSE for details.