Llama.cpp configuration #1793

so i actually dealt with this exact thing a while back when trying to plug llama.cpp into a similar setup, and yeah, that error is super annoying. the issue is likely how the system message is being handled in your custom endpoint. llama.cpp expects a very strict input format, and it sounds like the request payload you're sending might not be aligned with what it needs.

first, double-check your Provider config. when you're setting up the custom endpoint, make sure you're providing the exact expected fields in the POST request. for llama.cpp, you'd usually need something like a messages array with each message being an object that has role (system, user, or assistant) and content. if you're starting with API calls, the system message has to be the first thing in that array, or it'll throw that parser error you got.

example payload (not a code block, just writing it out):

{
  "messages": [
    { "role": "system", "content": "You're a helpful assistant." },
    { "role": "user", "content": "What's the weather today?" }
  ]
}

also, the way you're triggering this in the app matters. if you're passing a template or trying to use jinja for system messages, it's easy to mess up ordering. the error you posted seems to hint at that. maybe simplify the template down, or even hardcode the system message while testing.

are you testing with a small model like llama-7b? curious what endpoint setup you're using (local HTTP server? some proxy setup?). i've had to tweak headers and timeouts too in some cases, stuff like this can be finicky. let me know what you've tried so far!

Thanks so much for your help. I am actually just learning about how inference engines work and how to configure them locally. I understand what you are saying but have no ides where to configure some of the options you mentioned like the chat template. Interestingly the model such as gemma4-E4B fits on my RTX 4060 Ti 16GB card and I can use it with Cline in VS Code with no problem. It just doesn't work in OpenInterpreter. I can use either llama-server directly or llama-swap which opens up local Open AI API endpoints and I get the same result. I also don't know how to tweak headers and timeouts. If there is a guide you can point me to learn more about how to configure and run local models, it would be greatly appreciated.

@easygoer33 Based on the error and your follow-up, I don't think this is a problem with your RTX 4060 Ti or with llama.cpp itself.

The key part of the error is:

System message must be at the beginning.

That message is coming from the Jinja chat template used to format prompts before they are sent to the model. Many modern GGUF models (including Gemma, Llama 3/4, Qwen, etc.) rely on a specific chat template, and llama-server uses either the template embedded in the model or one you explicitly provide. The server documentation also exposes options such as --chat-template and --chat-template-file for overriding the template when needed. :contentReference[oaicite:0]{index=0}

Since you mentioned:

• ✅ the same model works in Cline
• ❌ the same model fails in Open Interpreter
• ❌ it fails whether you use llama-server directly or llama-swap

that strongly suggests the request generated by Open Interpreter isn't matching what the model's chat template expects, rather than a problem with your inference server.

A few things I'd try:

1. Verify the OpenAI-compatible endpoint first

   Make sure llama-server is responding correctly outside of Open Interpreter:

   curl http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model":"your-model",
       "messages":[
         {"role":"system","content":"You are a helpful assistant."},
         {"role":"user","content":"Hello"}
       ]
     }'

   If that succeeds, your server is probably configured correctly.

2. Check which chat template is being used

   Recent versions of llama.cpp support both built-in templates and custom Jinja templates via:

   --chat-template

   or

   --chat-template-file

   If you're using Gemma 4, newer chat templates have fixed several compatibility and tool-calling issues, so it's worth ensuring you're using an up-to-date template. :contentReference[oaicite:1]{index=1}

3. Compare the requests

   Since Cline works, I'd compare:

   • the JSON payload Cline sends
   • the JSON payload Open Interpreter sends

   If Open Interpreter inserts additional messages before the system message (or rearranges them), that would explain the exception.

From what I can tell, I couldn't find documentation showing that Open Interpreter exposes a setting to customize or override the Jinja chat template used by llama.cpp. The error also appears to originate from template rendering rather than from the HTTP endpoint itself. :contentReference[oaicite:2]{index=2}

If you're interested in learning more about local inference, I'd recommend starting with the official llama.cpp server documentation. It covers the OpenAI-compatible API, chat templates, Jinja support, and server configuration:

• tools/server/README.md
• llama-server --help

Those docs explain the purpose of options like --chat-template, --chat-template-file, and --jinja, which are the pieces most relevant to this error. :contentReference[oaicite:3]{index=3}

A couple of questions that would help narrow this down:

1. Which version of Open Interpreter are you using?
2. What exact command are you using to start llama-server (or llama-swap)?
3. Which GGUF model are you loading (for example, gemma-4-E4B-it from which publisher)?

Those details should make it possible to determine whether this is an Open Interpreter compatibility issue, a chat-template mismatch, or something specific to the model.

If this solves your problem, feel free to mark it as the accepted answer so others can find it easily.