gemserve

A lightweight OpenAI-compatible API server for Google's Gemma 4 models. No ollama, no vLLM — just transformers + torch served over FastAPI.

Defaults to google/gemma-4-E4B-it (4.5B effective params, BF16).

uv run serve_gemma4.py

That's it. The model downloads from HuggingFace on first run and the server starts at http://localhost:8000.

Point any OpenAI-compatible client at the server:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="gemma-4-E4B-it",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="")

Or with curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma-4-E4B-it",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

--model MODEL   HuggingFace model ID (default: google/gemma-4-E4B-it)
--host HOST     Bind address (default: 0.0.0.0)
--port PORT     Port (default: 8000)
--device DEVICE Device map (default: auto)

Example with a different model:

uv run serve_gemma4.py --model google/gemma-4-12B-it

• Python 3.11+
• GPU with sufficient VRAM for the model (E4B ~16GB in BF16, or CPU/GPU split with device_map="auto")
• uv (dependencies are inline in the script — no install step)

MIT