End-to-end tests for the geniex SDK, driven through the public Python binding so the suite doubles as a public-surface contract check.

tests/
├── assets/                # Real test images (quality_dog.jpg + NOTICE.md)
├── qdc/                   # QDC device dispatcher + per-platform entry scripts
├── test_api.py            # SDK metadata + resolve APIs (no model required)
├── test_llama_cpp.py      # llama_cpp plugin — LLM + VLM + precision + MTP
├── test_qairt.py          # qairt plugin     — LLM + VLM + precision
├── conftest.py            # Top-level fixtures (init, model paths, image)
├── pytest.ini             # Marker registry + suite discovery
├── _models.py             # Model-matrix loader for models.json
├── models.json            # The matrix: role -> [{id, precision, devices}]
└── _quality_data.py       # Keyword-quality prompts shared by both plugins

Every plugin file ships the same behavioural cases, plus an explicit model-manager-pull check that runs first:

Behavioural cases are parametrised over (model, device_map) pairs from models.json, so which backends run is a property of the model entry. Template / contract cases (ChatML sentinels, enable_thinking, tools rendering, mtmd markers) stay pinned to the role's first entry — they assert tokenizer specifics of one model. Model-manager pull failures are treated as FAIL, not SKIP, so a broken download surfaces as a red CI leg instead of a silent green skip.

Every generating cell passes GREEDY_TEMPERATURE (a negative argmax sentinel), not 0.0: both plugins read 0.0 as "unset" and substitute 0.8, so a pinned seed alone does not make a cell deterministic.

The conftest auto-tags items by location and device_map value:

snapdragon-marked and qairt items skip automatically unless GENIEX_DEVICE_TEST=1 is set and the host is a Snapdragon machine. QAIRT models are pulled on demand from AI Hub (like the llama_cpp models from HF), so the device shards need network access but no manual pre-pull.

# Anywhere — model-free API checks only
pytest tests -m api

# Anywhere — adds the llama_cpp CPU cells (downloads ~400 MB on first run)
pytest tests -m "api or (llama_cpp and device_cpu)"

# Snapdragon Windows ARM64 or Qualcomm Linux — full matrix
# (pulls every models.json entry, ~30 GB including gpt-oss-20b and the MTP pair)
GENIEX_DEVICE_TEST=1 pytest tests

tests/models.json is the matrix. Each role maps to a list of entries and every entry names the devices it runs on, so adding a model to the whole behavioural suite is a manifest edit. Loader: tests/_models.py.

Entry fields: id, precision, hub (default auto), devices, quality_max_new_tokens (per-model budget for the keyword cells), env_override (an env var that replaces id).

llama_cpp and QAIRT share the same LLM/VLM models so a keyword-quality divergence traces to backend / quantization rather than model identity. gpt-oss-20b is the hybrid-only entry and needs a bigger quality_max_new_tokens — its reasoning channel has no enable_thinking off switch.

The primary LLM is Qwen3-4B base, not Instruct-2507: Instruct-2507 emits a long <think> preamble before the answer that, on the 256-token budget the suite uses, pushes the keyword off the end of the completion and turns test_llm_quality_keywords into a thinking-budget test rather than a backend-quality test.

Swap models via env_override per entry, or GENIEX_TEST_MODELS for the whole matrix:

GENIEX_QAIRT_MODEL=qualcomm/<other-llm> \
GENIEX_QAIRT_VLM_MODEL=qualcomm/<other-vlm> \
GENIEX_DEVICE_TEST=1 pytest tests -m qairt

GENIEX_TEST_MODELS=/path/to/my-matrix.json GENIEX_DEVICE_TEST=1 pytest tests

The project splits into two test workflows:

• Unit Test (_unit-test.yml) — runs on every PR via pr-check.yml. Covers test-go, test-python, and test-sdk-ci (-m api) on linux-arm64 + windows-arm64 GitHub runners; no QDC hardware.

• QDC Test (_qdc-test.yml) — runs on workflow_dispatch and on v* tag push (via test.yml). Not on PR. One leg per platform; each leg runs the full plugin set (-m "llama_cpp or qairt"):

  Platform	Device	Framework
  Linux	QCS9075M	BASH
  Windows	SC8480XP	POWERSHELL

Each platform's entry script lives under qdc/: Linux uses the image's preinstalled python 3.12; Windows bootstraps python.org's arm64 embed zip. Both reuse sdk/benchmark/qdc/_qdc.py for submit / poll / log-collect.

Android (SM8850) is not wired yet — QDC's SM8850 image has neither python nor termux preinstalled, so it needs a CLI-driven harness that is out of scope for this iteration.

This directory is the home of SDK + plugin coverage. Tests under bindings/python/tests/ cover the binding layer itself (CLI wrapper, progress callbacks, model_manager Python surface, local pull paths) and do not launch real generation. Anything that reasons about device selection, plugin behaviour, or model output belongs here.