A tool for benchmarking and tracking Large Language Model (LLM) investment decisions.

This project provides a framework to create, manage, and track investment portfolios generated by LLM models. It allows you to:

• Create new portfolios
• List current holdings and recent context
• Update portfolios based on model decisions

The model executions and their current context can be seen here.

The active model roster is configured in models.json. The weekly GitHub Actions workflow runs the benchmark every Monday, writes model orders under orders/<model>/<date>.json, stores the market snapshot in prices/<date>.csv, updates llm100kbench.db, and regenerates this README's current portfolio section.

The project is intentionally limited to free API tiers. Models that require paid metered API access or subscriptions are archived instead of being run automatically.

Each weekly run also writes a concise decision log under logs/<model>/<date>.md, including the model used, validation status, per-trade rationale, context, and validation notes when a model response is rejected.

To optimize their portfolio, the primary objective defined for the LLMs, it is imperative to evaluate the risk-reward ratio, formulate cogent assumptions about future market conditions, and leverage tools and their understanding of human psychology and financial market dynamics.

This benchmark may be a good proxy to measure how well LLMs are able to coordinate the aforementioned efforts.

• chatgpt, deepseek, and grok are kept as continuing benchmark identities. Their exact backend model IDs are recorded in each new order's metadata.
• perplexity is archived for future runs because its API is paid and the free chat UI is not suitable for unattended automation.
• Claude and other paid-only APIs are not included while the project keeps the free-tier-only restriction.

• cmd: Contains the main command implementations
  • create: Initialize new portfolios
  • list: Display current holdings and context
  • update: Process investment orders and update holdings
  • stocks: Fetch most recent stock prices

The most recent prompt with the clear guidelines can be see here and here.

pie showData
    "deepseek" : 214020
    "chatgpt" : 123732
    "mistral" : 100000
    "qwen" : 100000
    "gpt-oss" : 95800
    "gemini" : 81862
    "llama" : 47102