Skip to content

GGUF source tests fail in sandboxed builds (Nix, Docker) #306

Description

@bgupta

Bug

Two tests added in v0.8.6 fail in any sandboxed build environment (Nix, Docker without network, CI with restricted egress):

  • models::tests::test_catalog_has_significant_gguf_coverage
  • models::tests::test_catalog_popular_models_have_gguf_sources

Error

thread 'models::tests::test_catalog_has_significant_gguf_coverage' panicked at llmfit-core/src/models.rs:1450:9:
GGUF source coverage is only 0.0% (0/1107), expected at least 25%

thread 'models::tests::test_catalog_popular_models_have_gguf_sources' panicked at llmfit-core/src/models.rs:1405:13:
Model meta-llama/Llama-3.3-70B-Instruct should have gguf_sources but has none

Root Cause

These tests expect the GGUF source catalog to be populated from HuggingFace at build time. In sandboxed environments (Nix builds, network-restricted Docker, offline CI), no network access is available, so the catalog is empty and the assertions fail.

Impact

  • nix build / nix profile install github:AlexsJones/llmfit fails on v0.8.6
  • Any CI pipeline that runs tests without network access will fail
  • v0.8.5 builds fine (tests didn't exist yet)

Suggested Fix

Gate these tests behind a feature flag or #[ignore] attribute so they don't run in sandboxed builds:

#[test]
#[ignore] // Requires network access to populate GGUF sources
fn test_catalog_has_significant_gguf_coverage() {

Or check for network availability at the start of the test and skip gracefully.

Workaround

In flake.nix, add checkFlags to skip the two tests:

checkFlags = [
  "--skip" "models::tests::test_catalog_has_significant_gguf_coverage"
  "--skip" "models::tests::test_catalog_popular_models_have_gguf_sources"
];

Environment

  • NixOS/nixpkgs unstable
  • llmfit v0.8.6
  • nix build with default sandbox enabled

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions