litert-ex

An Elixir/NIF wrapper around Google's LiteRT-LM C++ engine for on-device LLM inference. It can be used two ways:

• Embedded inference engine — call Litelm.RT.Native directly from your own Elixir/OTP app (no HTTP, no separate server process); the engine is linked in-process via the NIF, same as embedding LiteRT-LM in a native app.

• OpenAI-compatible HTTP server — run this project as a standalone server exposing LiteRT-LM over the OpenAI Chat Completions API, with an extension for server-side stateful chat sessions.

• NIF: c_src/litert_nif.cpp, built against LiteRT-LM's C API (c/engine.h)

• Embedded API: lib/litelm/litert_nif.ex — generate/1, create_session/1, send_message/2, delete_session/1

• HTTP: Plug.Router + Bandit, see lib/litelm/router.ex

• Endpoints: POST /v1/chat/completions, GET /v1/models, plus POST /v1/sessions / DELETE /v1/sessions/:id for stateful chat

• See Options reference for every accepted request option (model, temperature, top_p, top_k, seed, thinking, etc.) and Limitations for OpenAI request fields this project can't (or doesn't yet) support — notably n (multiple candidates), which isn't just unimplemented but currently unsupported by this engine build's CPU executor.

LM Studio is a good desktop app for using local models interactively, but it's a closed-source Electron GUI wrapping llama.cpp/MLX as a subprocess, strictly targeting desktop OSes. LiteRT-LM is a fundamentally different kind of thing — a linkable inference engine designed to reach far beyond the desktop:

• True cross-platform and mobile/web reach. LiteRT-LM runs natively on Android, iOS, and IoT/embedded targets, and in the browser via WebGPU — none of which LM Studio touches at all, since it's built exclusively for Mac/Windows/Linux desktops. The same LiteRT-LM source that this project builds as a server-side .so also ships in mobile apps and static web pages with no server or desktop wrapper involved.
• Deep NPU/GPU edge acceleration, not just desktop VRAM. Beyond CPU, LiteRT-LM targets mobile NPUs and a spread of accelerator backends (Metal, OpenCL, Vulkan, WebGPU) to get real performance and power efficiency on constrained edge hardware — a fundamentally different optimization target than LM Studio's desktop-GPU-or-CPU model. This project links the CPU-only build (//c:litert-lm-cpu, see c_src/litert_lm_cpu.BUILD.append) because that's what a Linux server needs, but the same engine source is what makes NPU-accelerated on-device inference possible elsewhere.
• In-app embedding, with no local server required. Because it's a C/C++ library (c/engine.h) meant to be linked directly into a host process — exactly what this project's NIF does via litert_lm_engine_create/litert_lm_conversation_send_message — an app can ship local AI entirely in-process. LM Studio's local server exists to serve its own app; it isn't designed to be embedded into someone else's mobile or web app at all.
• Advanced edge-native features: multi-modality and constrained decoding. LiteRT-LM has native support for vision/audio multi-modal input and constrained decoding (e.g. enforcing a JSON schema on generated output) — visible directly in its source tree, e.g. the constrained_decoding/gemma_model_constraint_provider build targets this project links against. These are first-class engine features aimed at on-device agent workflows, not something layered on top by a desktop chat UI.
• Apache-2.0 and source-available. LiteRT-LM can be vendored, patched, and rebuilt as part of this project's own build (c_src/Makefile clones and builds it from source, pinned to a specific tag). LM Studio's engine integration and app are closed-source; you consume it as-is.

Inference performance isn't one-sided — it depends on the hardware. On a desktop with a discrete GPU, LM Studio's llama.cpp/MLX backends are mature and heavily-optimized, and generally deliver competitive or faster raw token throughput and lower per-token latency there; that maturity on desktop GPUs is exactly what LM Studio is built to showcase. LiteRT-LM's performance advantage is edge-specific: on hardware llama.cpp doesn't specifically target — phone/IoT NPUs via vendor delegates, WebGPU in-browser — LiteRT-LM gets hardware acceleration and power-efficient inference LM Studio simply has no path to, because LM Studio never runs on that hardware at all. On a plain Linux CPU server like this project's target, neither engine has an inherent inference-speed edge over the other; the CPU-only build this project links (//c:litert-lm-cpu) is chosen for portability/embeddability, not because it's necessarily the fastest CPU inference path available — if raw CPU throughput on this exact box were the only goal, it would be worth benchmarking against llama.cpp directly rather than assuming either engine wins.

In short: LM Studio is a polished desktop app for poking at models yourself; LiteRT-LM is an embeddable, hardware-accelerated inference engine built to run everywhere from a server process (like this one) down to a phone's NPU or a browser tab — a reach LM Studio's desktop-only design can't match.

make            # fetch deps, build the NIF (clones + builds LiteRT-LM from
                # source the first time), compile the Elixir app

Run make help for the full target list (also delegates to c_src/'s own make help for NIF-build-specific targets/variables, e.g. overriding LITERT_LM_REF, MODEL_DIR, etc).

make get-model MODEL=gemma-4-E2B-it

Downloads into MODEL_DIR (default ~/.local/share/litelm/models). See make help for other known models and HUGGINGFACE_TOKEN (needed for gated repos like gemma-3n-*).

Add this project as a dependency and call Litelm.RT.Native directly from your own app — no HTTP server, no separate process. MODEL_DIR must still be set before the NIF loads (i.e. before your application starts), and a "model" value resolves to $MODEL_DIR/<model>.litertlm:

Payload maps use atom keys exclusively (%{model: "foo"}, not %{"model" => "foo"} — the latter is silently treated as a missing field, not an alternate spelling):

# stateless: re-prefills the full prompt every call
Litelm.RT.Native.generate(%{
  model: "gemma-4-E2B-it",
  prompt: "Hello",
  max_tokens: 64
})

# stateful: retains history + KV cache across turns in a NIF resource term
{:ok, session} = Litelm.RT.Native.create_session(%{model: "gemma-4-E2B-it"})
Litelm.RT.Native.send_message(session, %{content: "My favorite color is teal."})
Litelm.RT.Native.send_message(session, %{content: "What is my favorite color?"})
Litelm.RT.Native.delete_session(session)

# streaming: same session API, but delivers the reply incrementally as
# messages to the calling process instead of blocking for the full reply
{:ok, ref} = Litelm.RT.Native.send_message_stream(session, %{content: "Tell me a joke"})

receive_loop = fn receive_loop ->
  receive do
    {:litert_stream_chunk, ^ref, text} ->
      IO.write(text)
      receive_loop.(receive_loop)

    {:litert_stream_done, ^ref, _finalize} ->
      :ok

    {:litert_stream_error, ^ref, _reason, _finalize} ->
      :error
  end
end

receive_loop.(receive_loop)

See the moduledoc in lib/litelm/litert_nif.ex for the full payload/return shapes — in particular, send_message_stream/2's terminal messages carry a finalize field that's usually nil but must be checked and acted on (via finalize_pending_delete/1) if cancel_stream/1 or delete_session/1 is ever called while a stream is still in flight, or the session's memory leaks. This is the same underlying engine the HTTP server below is built on (Litelm.Server.ChatController and SessionRegistry are just a thin OpenAI-API layer over these same NIF calls) — use this mode when you want LiteRT-LM linked directly into your app's process instead of run as a standalone server.

MODEL_DIR=~/.local/share/litelm/models mix run --no-halt

The server listens on http://localhost:8000. GET /v1/models lists every *.litertlm file found in MODEL_DIR.

./scripts/chat --model gemma-4-E2B-it

scripts/chat gives you readline-style editing (history recall, Ctrl-R search, arrow keys) and a clean Ctrl-C quit. It's stateful by default (one server-side session for the whole conversation); pass --stateless to resend full history each turn instead. scripts/chat.exs is the same client without line editing, useful when piping input. See --help-style usage in either script's header, or just run one with no --model flag.

A real transcript (default stateful mode — POST /v1/sessions fires once at startup, then only the newest message is sent each turn; type a blank line to submit a message):

$ ./scripts/chat --model gemma-4-E2B-it

Chatting with 'gemma-4-E2B-it' at http://localhost:8000 (session 83f1173dd18cb5c2a8c31c5df8cfd76d). Type 'exit' to quit, 'reset' for a new session.

you> My favorite color is teal.
...>
assistant> That's a beautiful and unique choice! Teal is such a rich, versatile, and calming color.

What is it about teal that you love the most? ...

you> What is my favorite color?
...>
assistant> Your favorite color is **teal**.

you> exit

The second answer ("teal") is only correct because the server retained it via the session created at startup — scripts/chat sent nothing but "What is my favorite color?" on that request, no history. reset starts a fresh session (deletes the old one, creates a new one); exit deletes the current session before quitting.

aichat is a third-party OpenAI-compatible CLI client that works against this server as-is. A ready client config is checked in at config/aichat/config.yaml:

model: litertex:gemma-4-E2B-it
stream: false
clients:
  - type: openai-compatible
    name: litertex
    api_base: http://localhost:8000/v1
    models:
      - name: test-lm
        max_output_tokens: 32
      - name: gemma-4-E2B-it
        max_output_tokens: 64

stream: false is required here: aichat defaults to stream: true, but the config above has no session_id, so every request goes through the stateless path (POST /v1/chat/completions without session_id), which always replies with a single non-chunked JSON body regardless of that request field — this project only implements real SSE streaming for the stateful (session-based) path, not the stateless one (see Streaming below for why). Against a server that ignores "stream": true and returns plain JSON instead, aichat fails with Invalid response event-stream. If you drop this line, pass --no-stream/-S on every invocation instead. The stateful-sessions-with-aichat section below shows a config where stream: true does work correctly, because it routes through session_id.

Point aichat at it via AICHAT_CONFIG_DIR (so it doesn't touch your regular ~/.config/aichat/config.yaml) and run it, with the server already running and the model already downloaded (see above):

AICHAT_CONFIG_DIR=$(pwd)/config/aichat aichat

Or one-shot a single prompt:

AICHAT_CONFIG_DIR=$(pwd)/config/aichat aichat "What is the capital of France?"

aichat uses POST /v1/chat/completions in stateless mode (resends full history itself), so any model listed in models: above — or added there — just needs to match a .litertlm file's basename in MODEL_DIR. Add more models to the models: list as you download them with make get-model.

A few more of aichat's features that work against this server (all just AICHAT_CONFIG_DIR=$(pwd)/config/aichat aichat ..., omitted below for brevity):

• REPL mode — run aichat with no prompt argument to get an interactive Chat-REPL (tab autocompletion, history search, multi-line input) instead of a one-shot CMD. Since our server's /v1/chat/completions is stateless, aichat resends the growing conversation itself each turn — same mechanism as the CMD examples above, just kept alive across turns in one process.

  aichat
  > Hello!
  > What did I just say?

• Sessions (-s) — aichat's own client-side session tracking (saved under its config dir, independent of this server's /v1/sessions endpoint) so a conversation's context persists across separate CLI invocations, not just within one REPL process:

  aichat -s litertlm-demo "My favorite color is teal."
  aichat -s litertlm-demo "What's my favorite color?"

  Inside the REPL, the equivalent is .session litertlm-demo / .session (join/create) and .exit session (leave without deleting).
• Roles — bundle a system prompt (and optional model/config overrides) under a name, reusable across invocations:

  aichat --role %functions% --list-roles   # see built-ins, or define your own
  aichat -r translator "Translate to French: good morning"

  Custom roles are just .md files with an optional YAML frontmatter, under roles/ in AICHAT_CONFIG_DIR — e.g. config/aichat/roles/translator.md.
• File / stdin / URL input (-f) — feed local files, directories, URLs, or command output into the prompt (useful for testing a model's context handling against real documents, not just typed text):

  cat README.md | aichat "Summarize this file"
  aichat -f README.md "Summarize this file"

• --dry-run — render the message/prompt without actually calling the server. Does not print the raw outgoing HTTP request body (verified against aichat 0.30.0 — it only echoes the rendered input text), so it won't show whether a patch.body/AICHAT_PATCH_* field like session_id actually made it into the request; use the behavioral test in Using this server's stateful sessions with aichat instead (ask the server to recall something from a prior turn) to confirm a patched field is really reaching the server:

  aichat --dry-run "test prompt"

• --info / --list-models — confirm aichat picked up config/aichat/config.yaml correctly (right api_base, right models list) before troubleshooting anything server-side:

  aichat --info
  aichat --list-models

Two unrelated things are both called "sessions" here — neither description is wrong, they're just about different features: aichat's own docs are right that it natively supports sessions (-s/ .session, above) — that's real, and works against any provider. Separately, this server's /v1/sessions endpoint is a custom extension of this project, not part of the OpenAI Chat Completions spec — aichat has no idea it exists, no command or config field for it, because it wasn't built with this server in mind. The two are independent: aichat -s resends the entire conversation history itself on every request; this server's session_id mode expects exactly the opposite — one new message per request, with the server retaining history via the engine's KV cache. This section is entirely about the second kind.

To actually exercise this server's session_id-based sessions from aichat, inject session_id into the request body via aichat's per-model patch.body config field — this is the standard mechanism aichat provides for adding fields an OpenAI-compatible API doesn't natively define:

1. Create a session directly against the server first (aichat has no concept of creating this server's kind of session — you mint the id yourself, then hand it to aichat):

   SESSION_ID=$(curl -s http://localhost:8000/v1/sessions \
     -H 'content-type: application/json' \
     -d '{"model":"gemma-4-E2B-it"}' | jq -r .session_id)

2. Point a model entry's patch.body.session_id at it — this can be a throwaway config, since it's tied to one session's lifetime. stream: true works here (unlike the plain, session-less config above) since this server does implement real SSE streaming for the session-based path — verified: aichat renders the reply incrementally rather than erroring:

   mkdir -p /tmp/aichat-session-demo
   cat > /tmp/aichat-session-demo/config.yaml <<EOF
   model: litertex:gemma-4-E2B-it-session
   stream: true
   clients:
     - type: openai-compatible
       name: litertex
       api_base: http://localhost:8000/v1
       models:
         - name: gemma-4-E2B-it-session
           real_name: gemma-4-E2B-it
           max_output_tokens: 64
           patch:
             body:
               session_id: "$SESSION_ID"
   EOF

3. Every request aichat sends now carries "session_id" — the server routes it through SessionRegistry instead of the stateless path (see chat_controller.ex), which requires "messages" to contain exactly one user message per request (400s with too_many_messages otherwise). Plain aichat REPL mode (no -s) sends exactly one message per turn already — it keeps no history of its own between turns — so it's naturally compatible: launch the REPL against this config and chat normally, and the server is what remembers context across your turns, not aichat:

   AICHAT_CONFIG_DIR=/tmp/aichat-session-demo aichat
   > My favorite color is teal.
   That's a beautiful and calming color! ...
   > What is my favorite color?
   Your favorite color is teal.
   > .exit

   Verified interactively (via a real pty/tmux session, not piped input): the second turn's reply is correct only because the server retained "teal" via the session — confirmed by running the same two turns against a config with no session_id patch, where the second turn came back with no memory of the first at all.

   Don't combine this with -s/.session — that's aichat's own client-side session/history tracking, which resends every prior turn each request. Combined with a session_id that already has that history server-side, the second turn onward sends 2+ messages and immediately 400s with too_many_messages (confirmed by reproducing it: turn 1 succeeds, turn 2 fails with exactly that error). Use plain REPL (or one-shot CMD invocations, one message each) against a session_id-patched config instead — never -s with one.

4. Delete the session when done (it also expires on its own via SESSION_IDLE_TIMEOUT_MS, default 30 min):

   curl -s -X DELETE "http://localhost:8000/v1/sessions/$SESSION_ID"

This is a workaround, not native support — aichat has no built-in concept of a server that keeps its own session store, so nothing here lets aichat create/list/delete sessions itself; it's just carrying a fixed session_id on every request via patch.body, which is enough for a single ongoing conversation with a session you provisioned by hand.

The steps above edit patch.body.session_id in a YAML config file. aichat also supports patching the request body via an environment variable — AICHAT_PATCH_{CLIENT}_CHAT_COMPLETIONS, JSON-encoded, keyed by client name then model name — which does the exact same thing without writing a temp config, letting you pass SESSION_ID on the command line each time instead:

SESSION_ID=$(curl -s http://localhost:8000/v1/sessions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma-4-E2B-it"}' | jq -r .session_id)

AICHAT_PATCH_LITERTEX_CHAT_COMPLETIONS="{\"gemma-4-E2B-it\":{\"body\":{\"session_id\":\"$SESSION_ID\"}}}" \
  AICHAT_CONFIG_DIR=$(pwd)/config/aichat \
  aichat "My favorite color is teal."
# => That's a beautiful and unique choice! Teal is such a rich, versatile...

AICHAT_PATCH_LITERTEX_CHAT_COMPLETIONS="{\"gemma-4-E2B-it\":{\"body\":{\"session_id\":\"$SESSION_ID\"}}}" \
  AICHAT_CONFIG_DIR=$(pwd)/config/aichat \
  aichat "What is my favorite color?"
# => Your favorite color is **teal**.

LITERTEX in the env var name is this project's client name: from config/aichat/config.yaml — uppercased, matching aichat's own {CLIENT} placeholder convention. Verified live end-to-end: two separate one-shot aichat invocations (no -s, so aichat itself carries zero memory between them) — the second call only answers "teal" correctly because session_id reached the server and the server retained it.

Get the client name wrong and this fails silently — confirmed by testing AICHAT_PATCH_LITERT_CHAT_COMPLETIONS (missing the trailing EX in the LITERT client name) against the same setup: no error from aichat, session_id simply never reaches the request body, and the second turn comes back with no memory of the first (the same as any stateless request with no session_id at all) rather than a 4xx or a warning. If turn two doesn't recall context, double-check the env var name matches the client's name: field exactly before assuming the session itself is broken.

Same caveats as the config-file method apply: don't combine with -s/.session (see above), and the session still needs deleting when done (or it expires on its own via SESSION_IDLE_TIMEOUT_MS).

This is a general-purpose walkthrough of aichat itself — its own session/history feature (distinct from and unrelated to this server's /v1/sessions), plus roles, files, and macros. Everything here works against any provider aichat supports (OpenAI, Ollama, Claude, this project's server, etc.); the examples below assume whatever AICHAT_CONFIG_DIR/default config you already have set up, not this project's — swap in your own if needed.

A session is aichat maintaining a conversation's message history for you, resending it on every subsequent request so the model has full context — independent of, and unrelated to, this project's server-side session_id feature described above.

aichat -s demo "My favorite color is teal."
aichat -s demo "What is my favorite color?"
# => "Your favorite color is teal."

By default a session's history lives only in memory for that one aichat invocation and is discarded when it exits — --info shows save_session: null, meaning nothing is written to disk unless asked. Add --save-session (or set save_session: true in config) to persist it under sessions_dir (<AICHAT_CONFIG_DIR>/sessions/<name>.yaml) so it survives across separate invocations, verified above — the second command's reply ("teal") only came from a session file written by the first:

aichat -s demo --save-session "My favorite color is teal."
cat "$AICHAT_CONFIG_DIR/sessions/demo.yaml"      # plain YAML: model + messages

Inside the REPL, the equivalent flow is .session demo (create/join), chat normally, .info session to inspect it, .save session to persist it, and .exit session to leave without deleting. .empty session clears its messages without leaving it, and --list-sessions / .delete (interactive) manage saved sessions from outside/inside a session.

Caution mixing with a stateless custom server (like this one, absent the workaround above): a session resends its entire history every request — fine against a plain OpenAI-compatible stateless endpoint, but incompatible with any server that expects at most one message per request (this project's own session_id mode is exactly that — see above, and why -s can't be combined with it there).

A role is a reusable system prompt (optionally with its own model/param overrides), so you don't retype instructions every session:

mkdir -p "$AICHAT_CONFIG_DIR/roles"
cat > "$AICHAT_CONFIG_DIR/roles/pirate.md" <<'EOF'
You always answer as a pirate, using nautical slang and "arr".
EOF

aichat -r pirate "Say hello"
# => "Ahoy there, matey! ..."
aichat --list-roles     # shows pirate alongside aichat's built-ins

A role file is just a markdown file whose body is the system prompt; add YAML frontmatter (---\nmodel: ...\ntemperature: ...\n---) above it to pin a model or generation parameters to that role too. Inside the REPL: .role pirate to switch, .info role to inspect, .save role to persist edits, .exit role to leave it.

Feed local files, whole directories, remote URLs, or command output straight into the prompt — useful for asking about real content instead of retyping it:

aichat -f ./notes.txt "Summarize this"
aichat -f https://example.com "What is this page about?"
cat notes.txt | aichat "Summarize this"

Inside the REPL: .file notes.txt -- Summarize this, or .file %% to re-include the last reply as input for a follow-up.

A macro replays a fixed sequence of REPL commands under one name — useful for a repeatable multi-step setup (e.g. switch role, then ask a standard first question):

mkdir -p "$AICHAT_CONFIG_DIR/macros"
cat > "$AICHAT_CONFIG_DIR/macros/standup.yaml" <<'EOF'
steps:
  - .role pirate
  - What's the word for today, captain?
EOF

aichat --macro standup

Inside the REPL, .macro standup runs it directly, and --list-macros shows what's available.

aichat --rag <name> builds a local vector index over a set of documents (pdf, docx, url, etc.) so answers are grounded in that content rather than the model's own memory. This needs a provider with an embedding model configured (a client's models: entry with type: embedding, e.g. OpenAI's text-embedding-3-small) — creation fails with No available embedding model otherwise, since this project's own config/aichat/config.yaml doesn't declare one:

aichat --rag mydocs -f ./docs/         # first run: interactively prompts for
                                        # which embedding model to use, then indexes ./docs/
aichat --rag mydocs "What does the setup guide say about X?"

Inside the REPL: .rag mydocs to enter it, .sources rag to see which documents backed the last answer, .rebuild rag after adding/removing documents, --rebuild-rag from the CLI to do the same non-interactively, .exit rag to leave.

# stateless
curl -s http://localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma-4-E2B-it","messages":[{"role":"user","content":"Hello"}]}'

# stateful: create a session once, then send only the newest message each turn
SESSION_ID=$(curl -s http://localhost:8000/v1/sessions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma-4-E2B-it"}' | jq -r .session_id)

curl -s http://localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d "{\"model\":\"gemma-4-E2B-it\",\"session_id\":\"$SESSION_ID\",\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"

# stateful + streaming: same as above, plus "stream": true (only supported
# together with session_id — see Streaming below). -N/--no-buffer makes
# curl print each SSE event as it arrives instead of buffering the whole
# response before printing.
curl -sN http://localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d "{\"model\":\"gemma-4-E2B-it\",\"session_id\":\"$SESSION_ID\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}]}"

curl -s -X DELETE "http://localhost:8000/v1/sessions/$SESSION_ID"

POST /v1/chat/completions supports real, incremental streaming — via Server-Sent Events (Content-Type: text/event-stream, one chat.completion.chunk JSON object per data: ...\n\n event, terminated by a literal data: [DONE]\n\n) — matching OpenAI's own streaming format. Set "stream": true in the request body together with "session_id" (see curl above); streaming is only implemented for the stateful (session-based) path, not the stateless one. "stream": true without a session_id returns 400 streaming_requires_session rather than silently falling back to a buffered response.

A real transcript (curl -sN, session already created, MODEL_DIR pointing at a real gemma-4-E2B-it.litertlm — the plain SSE bytes after curl un-chunks the HTTP transfer-encoding for you):

$ curl -sN http://localhost:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d "{\"model\":\"gemma-4-E2B-it\",\"session_id\":\"$SESSION_ID\",\"stream\":true,\"messages\":[{\"role\":\"user\",\"content\":\"Say hello in one short sentence.\"}]}"

data: {"id":"chatcmpl-dbdbc3cc1abce31e","model":"gemma-4-E2B-it","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-dbdbc3cc1abce31e","model":"gemma-4-E2B-it","object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-dbdbc3cc1abce31e","model":"gemma-4-E2B-it","object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Each data: line arrives as its own TCP write as soon as that token/delta is ready — not buffered and flushed all at once — which is what makes this genuinely incremental rather than a single response dressed up in SSE framing (verified directly against the raw socket, not just curl's rendering of it). The first content-bearing chunk's delta carries "role":"assistant" (only that one does, per OpenAI's convention); subsequent chunks carry only delta.content; the final chunk has an empty delta and finish_reason: "stop", followed by the literal data: [DONE] terminator.

The same stream also works through aichat, using the session_id-patched config from stateful sessions with aichat above (that section's stream: true is this streaming path):

AICHAT_CONFIG_DIR=/tmp/aichat-session-demo aichat "Say hello in one short sentence."
# Hello! 😊

aichat prints the reply incrementally as data: ... chunks arrive over the wire — the same SSE stream as the curl transcript above, just rendered token-by-token by aichat instead of dumped as raw event lines. Remember stream: true only behaves this way against the stateful path here (session_id set); the plain stateless config (config/aichat/config.yaml, no session_id) sets stream: false for exactly this reason (see aichat above).

This is built on LiteRT-LM's own streaming C API (litert_lm_conversation_send_message_stream), exposed through the NIF as Litelm.RT.Native.send_message_stream/2 (see embedded inference engine above for calling it directly without HTTP at all) and wired into the router as the SSE response above. A client disconnecting mid-stream is detected and cancels the underlying generation server-side rather than letting it run to completion for nothing.

create_session/1/POST /v1/sessions accept an optional tools field — an OpenAI-shaped tools schema. Whether the model actually emits tool calls is model-dependent: only some model families (Gemma3/Gemma3N/Gemma4/FunctionGemma/Qwen3/Qwen2.5, auto-selected from the .litertlm file's own embedded metadata) parse tool calls out of generated text at all — gemma-4-E2B-it (the model this project's own make get-model downloads by default) does. Like thinking, this is stateful-path only and session-creation-time only — there's no tools field on the stateless path (generate/1/a POST /v1/chat/completions request without session_id gets a clear tools_requires_session error rather than silently ignoring it) and no per-turn override.

Tool execution happens entirely server-side, in Litelm.Server.ChatController — not the model, and not the HTTP client. When the model emits a tool call, ChatController.handle_chat/1 detects it, executes it itself via Litelm.Server.Tools, feeds the result back to the model as a role: "tool" turn, and repeats (up to 5 round trips) until the model produces a plain answer. The HTTP client only ever sees that final answer — intermediate tool calls and tool results never appear in the response.

Four built-in tools today:

• read_file — reads a file's contents, sandboxed to a configured root directory (READ_FILE_ROOT env var, default: the server's current working directory). Any path that resolves outside that root — via .. traversal or a directly-supplied absolute path elsewhere on disk — is a hard reject, no exceptions, no approval flow. This exists because a tool call's arguments are effectively untrusted input: the server executes them on the model's say-so, and the model's own output is itself derived from whatever a client's prompt talked it into requesting. Interactive out-of-root approval was considered and explicitly descoped — see Litelm.Server.Tools's moduledoc for why.
• write_file — writes text content to a file, creating parent directories as needed and overwriting an existing file without confirmation. Shares the exact same READ_FILE_ROOT sandbox as read_file — the sandbox boundary is the only protection this tool offers; there's no separate "don't overwrite" mode.
• list_directory — lists the names of files/subdirectories directly inside a given directory. Shares the exact same READ_FILE_ROOT sandbox as read_file (not a separate root/env var) — the three filesystem tools are meant to be used together (list, then read or write what looks relevant), and independently-configured boundaries would only add confusion without adding safety.
• execute_program — runs an external command and returns its stdout/stderr and exit status. Running an arbitrary model-chosen program with the server process's own privileges is a materially bigger risk than reading a sandboxed file, so this isn't path-sandboxed — it's restricted to a fixed allowlist of command names (EXECUTE_ALLOWED_COMMANDS env var, comma-separated, default: empty). This is opt-in, not opt-out: an unset or empty allowlist rejects every command rather than defaulting to permissive. There is deliberately no restriction on the arguments passed to an already-allowlisted command — execution goes through Elixir's System.cmd/3, which passes args to the OS as a real argv array, never interpolated into a shell string, so there's no shell-metacharacter injection risk to guard against, and second-guessing arguments to a command the operator already chose to allowlist wouldn't close any real gap. A non-zero exit status is not treated as an error at this layer — it's returned in the :ok result (%{output: ..., exit_status: n}) so the model can see and reason about it, the same way a real shell shows a caller.

{:ok, session_id} = Litelm.Server.SessionRegistry.create(
  model: "gemma-4-E2B-it",
  tools: [
    Litelm.Server.Tools.read_file_schema(),
    Litelm.Server.Tools.write_file_schema(),
    Litelm.Server.Tools.list_directory_schema(),
    Litelm.Server.Tools.execute_program_schema()
  ]
)

# "Read this file and summarize it" now genuinely reads a real file on
# disk (within READ_FILE_ROOT) and the answer reflects its actual
# contents -- verified directly: a session asked to read a fixture file
# containing a random marker string correctly quoted that string back,
# something the model has no way to produce except via a real file read.
{:ok, response} = Litelm.Server.ChatController.handle_chat(%{
  "session_id" => session_id,
  "messages" => [%{"role" => "user", "content" => "Read notes.txt and summarize it."}]
})

Over HTTP, the same thing is just "tools": [...] in the POST /v1/sessions body:

curl -s http://localhost:8000/v1/sessions \
  -H 'content-type: application/json' \
  -d '{"model":"gemma-4-E2B-it","tools":[{"type":"function","function":{"name":"read_file","description":"Read a file","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}}]}'

execute_program needs its allowlist configured before it's usable — e.g. EXECUTE_ALLOWED_COMMANDS=ls,git mix run --no-halt (or set on the server's environment however you normally configure MODEL_DIR).

No special aichat support is needed — tool calling reuses the exact same session_id-via-patch.body mechanism already covered above; tools is just another field on the same POST /v1/sessions call that mints the session aichat then carries on every request. Create the session with tools set, patch its session_id into aichat the same way (config file or the AICHAT_PATCH_*_CHAT_COMPLETIONS env var), and aichat needs to do nothing tool-aware at all — the model emits a tool call, this server executes it and re-prompts internally (see above), and aichat only ever sees the final plain-text answer, same as any other reply:

TOOLS='[{"type":"function","function":{"name":"read_file","description":"Read the contents of a file at the given path, relative to the server'\''s configured root directory.","parameters":{"type":"object","properties":{"path":{"type":"string"}},"required":["path"]}}}]'

SESSION_ID=$(curl -s http://localhost:8000/v1/sessions \
  -H 'content-type: application/json' \
  -d "{\"model\":\"gemma-4-E2B-it\",\"tools\":$TOOLS}" | jq -r .session_id)

AICHAT_PATCH_LITERTEX_CHAT_COMPLETIONS="{\"gemma-4-E2B-it\":{\"body\":{\"session_id\":\"$SESSION_ID\"}}}" \
  AICHAT_CONFIG_DIR=$(pwd)/config/aichat \
  aichat "Read the file at path test/fixtures/secret.txt and tell me what it says."
# => The file `test/fixtures/secret.txt` contains the following text:
#    "The secret code is: PINEAPPLE-99"

Verified live end-to-end against gemma-4-E2B-it: the model correctly quoted a fixture file's contents back — something it has no way to produce except via a real, server-executed read_file call — and aichat never had to know a tool call happened at all. aichat's own, separate function-calling feature (--role %functions%) is unrelated and not needed here — tools are declared once at session-creation time, not per-aichat-invocation.

Running the steps above by hand every time (create session, patch the env var, remember to delete the session after) gets old fast — scripts/liteai does it in one command: creates the session (all four built-in tools by default), sets AICHAT_PATCH_LITERTEX_CHAT_COMPLETIONS and AICHAT_CONFIG_DIR for you, runs aichat (REPL or one-shot, whatever you pass through), and deletes the session again on exit (normal exit, exit/quit/Ctrl-D, or an error):

MODEL_DIR=~/.local/share/litelm/models EXECUTE_ALLOWED_COMMANDS=echo,ls mix run --no-halt &

scripts/liteai "Read the file at path test/fixtures/secret.txt and tell me what it says."
# => liteai: session <id> (gemma-4-E2B-it, tools: read_file,write_file,list_directory,execute_program)
#    The file `test/fixtures/secret.txt` says: "The secret code is: PINEAPPLE-99".

scripts/liteai --tools read_file,list_directory   # REPL, only those two tools
scripts/liteai --no-tools "plain chat, no tools declared at all"
scripts/liteai --model test-lm --host http://localhost:9000 "..."

The tool schemas it declares are fetched live from Litelm.Server.Tools (via mix run --no-start -e ...), not duplicated in the script, so it can never drift out of sync with the real ones. It only controls which tool schemas are offered to the model — execute_program still needs EXECUTE_ALLOWED_COMMANDS set on the server's own environment (as in the example above) before it can actually run anything.

See Litelm.RT.Native's moduledoc "Tool calling" section for the full tool_calls/send_tool_result/2 payload shapes if calling the NIF directly (embedded mode) rather than through the HTTP server, and Litelm.Server.Tools's moduledoc for the sandboxing details. Streaming (send_message_stream/"stream": true) does not support tool calling in this version — see Limitations.

Every request-level option this project accepts, across both the embedded API (Litelm.RT.Native, atom-keyed payload maps) and the HTTP server (POST /v1/chat/completions stateless, POST /v1/sessions stateful — string-keyed JSON). Sampling parameters (temperature, top_p, top_k, seed) are set once per session/one-shot call, never per-turn — send_message/2/send_message_stream/2 and a session-based chat turn only accept max_tokens as a per-turn override.

These aren't two alternative sampling algorithms to pick between (as they're sometimes presented) — LiteRT-LM's TOP_P sampler (the only sampler type this build implements; see Limitations) applies both in sequence:

1. Truncate the vocabulary to the top_k highest-probability tokens.
2. Within that already-truncated set, keep only the smallest prefix (by probability, highest first) whose cumulative probability reaches top_p, then sample from what's left.

So top_k is a hard ceiling on candidate pool size, and top_p narrows further inside it. The practical consequence for this project's default (top_k: 1): the candidate pool is exactly one token no matter what top_p, temperature, or seed are set to, so sampling is effectively deterministic (argmax-like) by default. seed/top_p only have any visible effect once top_k is raised above 1 — verified directly: with top_k: 1, five different seed values produced identical output; with top_k: 40, top_p: 0.98, they produced five genuinely different completions.

A few OpenAI Chat Completions parameters are either not implemented or not implementable against this engine build, rather than just "not wired up yet":

• n (multiple candidates) is not supported at all, on any path. SessionConfig::SetNumOutputCandidates exists in LiteRT-LM's C++ API and looks like the right hook, but this build's CPU executor hardcodes batch size 1 (RET_CHECK_EQ(batch_size, 1) << "Only support batch size 1 for now." in llm_litert_compiled_model_executor.cc) — confirmed by actually requesting num_output_candidates: 3 against both a real model (gemma-4-E2B-it) and the tiny synthetic test-lm fixture used in this project's own test suite; both fail GenerateContent with a generic error every time. Not something this project can fix without an upstream LiteRT-LM change; there's no request field for it, and one won't be added until batch size > 1 actually works.
• top_p, top_k, and seed are supported; frequency_penalty/ presence_penalty and custom stop sequences are not. generate/1 and create_session/1 both accept optional top_p (default 0.95), top_k (default 1), and seed fields — same via /v1/chat/completions (stateless path) and POST /v1/sessions (session-level, set once at creation, not per-turn — matching how temperature already works). seed genuinely influences the sampling outcome (verified: different seeds with a wide top_k/top_p and nonzero temperature produce different text), but is not a bit-exact reproducibility guarantee across separate calls — the same seed can still produce different output between two calls, almost certainly floating-point non-associativity from multi-threaded CPU matmul execution rather than the seed not reaching the sampler (num_threads is a real, separately configurable LlmExecutorSettings field, not yet exposed by this NIF). There's still no frequency_penalty/presence_penalty setter surfaced, and no custom stop sequence override beyond the model's own fixed BOS/EOS tokens.
• reasoning_content (from create_session/1's thinking: true option — see embedded inference engine above) is stateful-path only. There's no equivalent for the stateless /v1/chat/completions path (no session_id): generate/1's underlying Session::GenerateContent call never builds the Conversation/prompt- template machinery that thinking depends on. There's also no universal channel name across models — whether thinking: true actually produces separate reasoning content (versus just being a no-op) depends entirely on whether the specific .litertlm file's own metadata defines a channel for it.
• Function calling is supported, but stateful-path only and without streaming. See Tool calling above — create_session/1's tools field, model-dependent (only Gemma3/Gemma3N/Gemma4/ FunctionGemma/Qwen3/Qwen2.5 processors parse tool calls out of generated text at all), server-executed (Litelm.Server.Tools, four built-in tools — read_file/write_file/list_directory sandboxed to a configured root, execute_program restricted to an allowlist of command names), with no equivalent on the stateless path or send_message_stream/"stream": true yet.
• Structured outputs (response_format: json_schema) and logprobs/top_logprobs are unimplemented. LiteRT-LM's engine has a general-purpose constrained-decoding mechanism (ConversationConfig:: Builder::SetEnableConstrainedDecoding/OptionalArgs.decoding_constraint — the same underlying feature tool-call parsing could optionally use for guaranteed-valid JSON, though this project doesn't enable it even for tools; see Tool calling) that isn't wired to response_format here, and a text-scoring API that could back logprobs but isn't exposed either.
• A second turn against a session whose conversation has already grown past the model's context window (KV-cache ceiling) crashes the whole BEAM VM — a genuine, open engine bug, not fixable from this project's side. create_session/1 opts in to LiteRT-LM's own ConversationConfig::Builder::SetReturnErrorOnMaxTokensReached(true), so hitting the ceiling is meant to surface as a clean {:error, :context_window_exceeded} (see Litelm.RT.Native's moduledoc "Errors" section) rather than the silent truncation that was this project's prior default behavior. In practice, though, once a session has reached that state, the next send_message/2/send_tool_result/2 call against the same session segfaults inside LiteRT-LM's own AVX2 SIMD decode path on a background execution thread — confirmed via dmesg, and confirmed independent of the SetReturnErrorOnMaxTokensReached opt-in itself (reproduces identically, just via SIGABRT instead of SIGSEGV, with that flag temporarily removed). See test/litelm/context_exhaustion_crash_test.exs (tagged :known_crash, excluded even from --include integration — run explicitly with mix test --include known_crash) for a live reproduction and the reasoning behind why it's structured as a tripwire rather than a "prove it's fixed" test. Litelm.Server.AnthropicSessionCache.evict/1 is this project's only available mitigation: it exists specifically so a caller reacting to :context_window_exceeded on the first failing call never sends a second message to that same poisoned session — but there is no way to prevent the crash from inside this NIF if a caller's own retry logic (or a client resending an oversized conversation from scratch against a fresh session) triggers it a different way.

mix test                        # unit tests only (fast, no model required)
MODEL_DIR=~/.local/share/litelm/models mix test --include integration
                                 # also runs NIF/model-backed integration tests