Skip to content

Questions about sharing benchmarks #862

Description

@datawookie

Bug description

I'm trying to share a benchmark.

$ llmfit bench --share 

  The following stored benchmark data would be contributed:

    - gemma3:4b via ollama — 7.9 tok/s
    - gemma3:4b via ollama — 8.9 tok/s
    - qwen3.5 via llamacpp — 1.4 tok/s
    - muse-glimmer via llamacpp — 1.4 tok/s

  Share all 4 local benchmark submission(s) as a PR to AlexsJones/llmfit? [y/N] y
  GitHub: authenticated as datawookie
  Share failed: GitHub API returned 404: Not Found
  Your results remain stored locally; retry with `llmfit bench --share`.

Two things I wanted to raise:

  1. The 404 error.
  2. Is is possible to just share one benchmark? For example, right now I only want to upload the muse-glimmer result.

Expected behavior

  1. No 404 error.
  2. Ability to select from local benchmarks to upload.

Actual behavior

  1. 404 error.
  2. Looks like all with be uploaded.

Steps to reproduce

llmfit bench --share

Diagnostic report (llmfit doctor)

The following stored benchmark data would be contributed:

    - gemma3:4b via ollama — 7.9 tok/s
    - gemma3:4b via ollama — 8.9 tok/s
    - qwen3.5 via llamacpp — 1.4 tok/s
    - muse-glimmer via llamacpp — 1.4 tok/s

  Share all 4 local benchmark submission(s) as a PR to AlexsJones/llmfit? [y/N] y
  GitHub: authenticated as datawookie
  Share failed: GitHub API returned 404: Not Found
  Your results remain stored locally; retry with `llmfit bench --share`.

llmfit version

llmfit 1.16

Operating system

Linux

Affected component

CLI output (subcommands, table display)

Mode used

CLI (--cli flag)

GPU details (if relevant)


Additional context

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions