TMIQ: Quantifying Test and Measurement Domain Intelligence in Large Language Models

TMIQ (Test and Measurement Intelligence Quotient) is a benchmark and toolkit designed to quantitatively assess the capabilities of Large Language Models (LLMs) within the specialized domain of Test and Measurement, particularly focusing on electronic engineering tasks. This project provides tools for generating domain-specific datasets, evaluating LLMs against these datasets, and analyzing their performance across various metrics including accuracy, cost, and efficiency.

The Test and Measurement industry increasingly adopts Generative AI, and LLMs show promise for automation and precision. However, evaluating LLMs in this specialized area requires domain-specific benchmarks. TMIQ addresses this gap by offering a comprehensive suite for assessing LLM performance on tasks like generating Standard Commands for Programmable Instruments (SCPI), understanding electronic engineering concepts, and performing structured reasoning.

• Domain-Specific Evaluation: Focuses on electronic engineering and Test and Measurement tasks.
• Comprehensive Metrics: Evaluates LLMs on SCPI command generation (accuracy, partial matching), ranked response evaluation (First Match, Position Match), and operational efficiency (cost, time, token usage).
• Synthetic Dataset Generation: Tools to create custom datasets from source documents (e.g., PDF manuals converted to Markdown) using LLMs for question and distractor generation.
  • bookmine (construct.py): Generates question-answer pairs from technical documents.
  • falsemine (falsemine.py): Generates plausible false answers (distractors) for multiple-choice questions.
• Multiple LLM Support: Integrates with various LLM providers including OpenAI, Anthropic (via Bedrock), Google (Vertex AI Gemini), Mistral (via Bedrock), Groq, and local Hugging Face TGI instances. See models.csv for a list of pre-configured models.
• Contextual & Non-Contextual Evaluation: Supports evaluating LLMs with or without domain-specific contextual information (e.g., for SCPI command generation).
• Result Analysis & Visualization: Generates detailed plots (correctness, cost, time, token usage, efficiency) and LaTeX tables for easy comparison and reporting.
• Command-Line Interface: Provides a user-friendly CLI (tmiq) for managing datasets, running benchmarks, and generating reports.
• Reproducibility: Offers tools to download standard datasets and generate custom ones using a defined methodology.

• Dataset Generation (construct.py, falsemine.py, download.py):
  • Download pre-defined benchmark datasets (EEMT, SCPI, MMLU-Pro subsets).
  • Generate new question-answer pairs from your technical documents.
  • Create multiple-choice questions by generating distractor answers.
• LLM Interaction (models.py):
  • A unified interface for querying different LLM APIs.
  • Handles API-specific request formatting, retries, token counting, and cost estimation.
• Evaluation Engine (evaluator.py):
  • Orchestrates the benchmarking process for different task types (e.g., EEMT multiple-choice, SCPI command generation).
  • Constructs prompts, sends them to LLMs, and extracts answers.
• Scoring Metrics (utils/metrics.py, check.py):
  • Implements various metrics like First Match Score, Position Match Score, SCPI Command Accuracy, SCPI Score (partial match), and overall efficiency.
• Plotting and Reporting (plot.py):
  • Generates visual comparisons of LLM performance, cost, and efficiency.
  • Produces LaTeX tables summarizing benchmark results.
• CLI (tmiq_cli.py):
  • The main entry point to access all TMIQ functionalities.

tmiq_prod
├── models.csv                      # CSV listing supported LLMs, their IDs, and costs
├── README.md                       # This README file
├── justfile                        # Task runner file for common commands
├── tmiq                            # Main Python package directory
│   ├── evaluator.py                # Core evaluation logic for different benchmarks
│   ├── download.py                 # Script to download datasets
│   ├── construct.py                # (bookmine) Generates Q/A pairs from documents
│   ├── falsemine.py                # Generates false/distractor answers
│   ├── models.py                   # LLM API client implementations
│   ├── plot.py                     # Generates plots and LaTeX tables from results
│   ├── tmiq_cli.py                 # Main CLI entry point
│   ├── init_dir.py                 # Initializes a project directory
│   ├── upload.py                   # Uploads datasets to Hugging Face
│   ├── view.py                     # View markdown file structure/word counts
│   ├── SCPI.py                     # SCPI command extraction from markdown
│   ├── config/                     # Default YAML configuration files
│   ├── scripts/                    # Utility scripts (e.g., for local TGI server)
│   └── utils/                      # Utility modules (metrics, preprocessing, debug)
├── test                            # Test configurations and scripts (example)
├── .gitignore
└── setup.py                        # Package setup script

• Python 3.8+
• pip package installer

1. Clone the repository (if not installing from PyPI):

   git clone http://github.com/labiium/tmiq
   cd tmiq_prod

2. Create and activate a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install TMIQ and its dependencies:

   pip install .

   Alternatively, if you only cloned and want to install requirements:

   pip install -r requirements.txt # (Create requirements.txt via pip freeze > requirements.txt if needed)

4. Set up Environment Variables: Some LLM providers require API keys. Create a .env file in the root of the project directory (tmiq_prod/) and add your keys:

   OPENAI_API_KEY="your_openai_api_key"
   GROQ="your_groq_api_key"
   GCLOUD_PROJECT="your_gcloud_project_id"
   # AWS credentials should be configured as per Boto3 documentation (e.g., via ~/.aws/credentials or environment variables)

   The tmiq/models.py script loads these variables.

TMIQ provides a command-line interface. The general usage is:

tmiq <command> [options]

You can see help for any command:

tmiq <command> -h

1. init: Initialize a project directory with default configurations and scripts.

   tmiq init --directory my_tmiq_project
   cd my_tmiq_project

   This will create config/ and scripts/ subdirectories with example files.

2. download: Download standard benchmark datasets.

   tmiq download --directory .data

   This downloads datasets like EEMT, SCPI, MMLU-Pro subset, etc., into the .data directory.

3. bookmine (Dataset Generation - Questions & Answers): Generate question-answer pairs from Markdown documents (e.g., converted from PDF manuals).

   # First, convert PDFs to Markdown (e.g., using marker)
   # marker extract path/to/pdfs/ path/to/markdown_output/
   
   # Then run bookmine with a configuration file
   tmiq bookmine config/construct_example.yaml

   The construct_example.yaml (or your custom config) specifies the input Markdown files, the LLM to use for generation, output schema, and other parameters.

4. falsemine (Dataset Generation - Distractors): Generate false/distractor answers for existing question-answer pairs to create multiple-choice questions.

   tmiq falsemine config/falsemine_example.yaml

   The falsemine_example.yaml specifies the input dataset, LLM for generation, number of false answers, etc.

5. run: Execute a benchmark.

   tmiq run <config_path> <model_identifier> <evaluator_name>

   • config_path: Path to the evaluator's YAML configuration file (e.g., config/eemt_eval_config.yaml). This file specifies the dataset to use and prompting strategy.
   • model_identifier: The ID of the LLM to test. Refer to models.csv for available model IDs and their corresponding providers (e.g., gpt-4o-mini-2024-07-18, anthropic.claude-3-haiku-20240307-v1:0).
   • evaluator_name: The type of benchmark to run.
     • eemt: Electronics Engineering Multiple-choice Test.
     • scpi: SCPI command generation test.
     • mmlu: MMLU-Pro subset evaluation.

   Example:

   tmiq run config/eemt_eval_config.yaml gemini-1.5-flash-001 eemt
   tmiq run config/scpi_eval_config.yaml gpt-4o-mini-2024-07-18 scpi --debug_level DEBUG

   Results (Parquet files and JSON summaries) will be saved to the output directory specified in the evaluator's config.

6. plot: Generate plots and LaTeX tables from benchmark results.

   tmiq plot config/plot_config_example.yaml

   The plot_config_example.yaml specifies the type of plot, input directories containing result Parquet files, titles, and the output directory for generated plots/tables.

7. upload: Upload a Parquet dataset to the Hugging Face Hub.

   tmiq upload path/to/your_dataset.parquet your_hf_username/dataset_name

8. view: Analyze a Markdown file and print word counts per section.

   tmiq view path/to/document.md --level 2

A justfile is provided for convenience to run common tasks. If you have just installed:

just # Lists available recipes
just build
just test-benchmark
just plot_eemt

The tmiq/scripts/tgi.sh script can be used to start a Hugging Face Text Generation Inference (TGI) server locally for models like Llama. You would then configure tmiq to use the local TGI endpoint (usually by setting the route in a custom HuggingfaceClient or LLamaClient instantiation if not already supported by default).

This benchmark evaluates LLMs on multiple-choice questions covering various electronic engineering topics. Performance is measured by the ability to correctly rank or select the right answer from a list of options (including generated distractors).

This task assesses an LLM's ability to generate accurate Standard Commands for Programmable Instruments (SCPI). Evaluation includes:

• Exact match accuracy: Whether the generated command precisely matches the correct SCPI command.
• SCPI Score (Partial match): Awards partial credit based on the closeness of the generated command to the correct one, considering abbreviations and argument correctness.
• Contextual Evaluation: Can be run with or without providing the LLM with the instrument's programming manual (SCPI command list) as context.

TMIQ employs a range of metrics to provide a holistic view of LLM performance:

• Correctness:
  • FirstMatchScore: Awards full marks if the highest-ranked/first-selected answer is correct.
  • PositionMatchScore: Awards exponentially decaying score based on the position of the correct answer in the LLM's ranked list.
  • ExactMatch: For tasks requiring precise output, checks for an exact match.
  • SingeltonMatchScore: For single answer extraction tasks.
  • ContainedMatchScore (for SCPI): Checks if the generated command is one of the valid SCPI commands.
  • SCPIMatchScore: A detailed partial matching score for SCPI commands, considering command hierarchy, abbreviations, and arguments.
• Efficiency:
  • Cost: Total API cost in USD for running the benchmark (calculated from input/output token counts and provider rates in models.csv).
  • Time: Total execution time for the benchmark.
  • Token Usage: Total number of input and output tokens consumed.
  • Overall Efficiency Metric (E): A composite score combining normalized accuracy, cost, and time, calculated as (S_norm * C_norm * T_norm)^(1/3).

TMIQ uses YAML files for configuring various operations like dataset generation (bookmine, falsemine), benchmark runs (run), and plotting (plot). Example configuration files are provided in the config/ directory (or created by tmiq init).

• Model Configuration: models.csv lists available LLMs, their API identifiers (slug), provider (source), and token costs.
• Dataset Generation Configs: Define input paths, LLMs for generation, prompt templates, number of items, output schemas, etc.
• Evaluator Configs: Define the dataset file, base prompts, example formats, and output directory for results.
• Plotting Configs: Define input result directories, plot types, titles, and output locations.