Introduce initial mcp-bench Python package with MCP benchmarking API and CLI - #1

This PR bootstraps mcp-bench as a pip-installable package with both a Python API and a CLI for benchmarking MCP-driven tool-calling via OpenAI-compatible endpoints (including local Ollama). It adds structured benchmark outputs that capture full run traces (chat history, tool calls, arguments, and tool results), plus repeat/temperature experiment controls and dataframe conversion utilities.

• Packaging and distribution

  • Added pyproject.toml-based packaging (mcp-bench), src/ layout, and CLI entrypoint:
    • mcp-bench = mcp_bench.cli:main
  • Added optional dataframe extra dependency:
    • mcp-bench[dataframe] → pandas

• Core benchmarking API

  • Added MCPConfig + load_mcp_config(...) to load MCP JSON configs and normalize function/tool specs into OpenAI tool format.
  • Added MCPBenchRunner with:
    • run_prompt(prompt, repeats, temperature)
    • run_experiment(prompts, repeats, temperature)
  • Added structured result models:
    • PromptRunResult (prompt/model/temperature/final response/chat history/tool calls)
    • ToolCallRecord (function name/arguments/result)
  • Added robust handling for malformed tool-call argument JSON and non-JSON-serializable tool results.

• CLI surface

  • Added CLI for running experiments from MCP config:
    • --config, --model, repeated --prompt, --repeats, --temperature, --base-url, --api-key, --output
  • Emits benchmark results as JSON (stdout or file).

• Dataframe utility

  • Added results_to_dataframe(results) to convert benchmark runs into analysis-friendly tabular output, including serialized tool-call records.

from mcp_bench import MCPBenchRunner, load_mcp_config, results_to_dataframe

config = load_mcp_config("mcp-config.json")
runner = MCPBenchRunner(model="llama3.1", config=config)
results = runner.run_prompt("What is 1+2?", repeats=5, temperature=0.2)
df = results_to_dataframe(results)