Bug
Two tests added in v0.8.6 fail in any sandboxed build environment (Nix, Docker without network, CI with restricted egress):
models::tests::test_catalog_has_significant_gguf_coverage
models::tests::test_catalog_popular_models_have_gguf_sources
Error
thread 'models::tests::test_catalog_has_significant_gguf_coverage' panicked at llmfit-core/src/models.rs:1450:9:
GGUF source coverage is only 0.0% (0/1107), expected at least 25%
thread 'models::tests::test_catalog_popular_models_have_gguf_sources' panicked at llmfit-core/src/models.rs:1405:13:
Model meta-llama/Llama-3.3-70B-Instruct should have gguf_sources but has none
Root Cause
These tests expect the GGUF source catalog to be populated from HuggingFace at build time. In sandboxed environments (Nix builds, network-restricted Docker, offline CI), no network access is available, so the catalog is empty and the assertions fail.
Impact
nix build / nix profile install github:AlexsJones/llmfit fails on v0.8.6
- Any CI pipeline that runs tests without network access will fail
- v0.8.5 builds fine (tests didn't exist yet)
Suggested Fix
Gate these tests behind a feature flag or #[ignore] attribute so they don't run in sandboxed builds:
#[test]
#[ignore] // Requires network access to populate GGUF sources
fn test_catalog_has_significant_gguf_coverage() {
Or check for network availability at the start of the test and skip gracefully.
Workaround
In flake.nix, add checkFlags to skip the two tests:
checkFlags = [
"--skip" "models::tests::test_catalog_has_significant_gguf_coverage"
"--skip" "models::tests::test_catalog_popular_models_have_gguf_sources"
];
Environment
- NixOS/nixpkgs unstable
- llmfit v0.8.6
nix build with default sandbox enabled
Bug
Two tests added in v0.8.6 fail in any sandboxed build environment (Nix, Docker without network, CI with restricted egress):
models::tests::test_catalog_has_significant_gguf_coveragemodels::tests::test_catalog_popular_models_have_gguf_sourcesError
Root Cause
These tests expect the GGUF source catalog to be populated from HuggingFace at build time. In sandboxed environments (Nix builds, network-restricted Docker, offline CI), no network access is available, so the catalog is empty and the assertions fail.
Impact
nix build/nix profile install github:AlexsJones/llmfitfails on v0.8.6Suggested Fix
Gate these tests behind a feature flag or
#[ignore]attribute so they don't run in sandboxed builds:Or check for network availability at the start of the test and skip gracefully.
Workaround
In
flake.nix, addcheckFlagsto skip the two tests:Environment
nix buildwith default sandbox enabled