Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

changelog : llama-server REST API #9291

Open
ggerganov opened this issue Sep 3, 2024 · 3 comments
Open

changelog : llama-server REST API #9291

ggerganov opened this issue Sep 3, 2024 · 3 comments
Labels
documentation Improvements or additions to documentation

Comments

@ggerganov
Copy link
Owner

ggerganov commented Sep 3, 2024 •
edited
Loading

Overview

This is a list of changes to the public HTTP interface of the llama-server example. Collaborators are encouraged to edit this post in order to reflect important changes to the API that end up merged into the master branch.

If you are building a 3rd party project that relies on llama-server, it is recommended to follow this issue and check it carefully before upgrading to new versions.

See also:

Recent API changes (most recent at the top)

version PR desc
TBD. #9865 Add option to time limit the generation phase
TBD. #9860 Remove self-extend support
b3910 #9857 Remove legacy system prompt support
b3897 #9776 Change default security settings, /slots is now disabled by default
Endpoints now check for API key if it's set
b3887 #9510 Add /rerank endpoint
b3754 #9459 Add [DONE]\n\n in OAI stream response to match spec
b3721 #9398 Add seed_cur to completion response
b3683 #9308 Environment variable updated
b3599 #9056 Change /health and /slots

For older changes, use:

git log --oneline -p b3599 -- examples/server/README.md

Upcoming API changes

  • TBD
@ggerganov ggerganov added the documentation Improvements or additions to documentation label Sep 3, 2024
@ggerganov ggerganov pinned this issue Sep 3, 2024
@ggerganov ggerganov mentioned this issue Sep 3, 2024
Open
@ngxson
Copy link
Collaborator

ngxson commented Sep 7, 2024

Not a REST API breaking change, but is server-related: some environment variables are changed in #9308

@slaren
Copy link
Collaborator

slaren commented Sep 13, 2024

After #9398, in the completion response seed contains the seed requested by the user, while seed_cur contains the seed used to generate the completion. The values can be different if seed is LLAMA_DEFAULT_SEED (or -1), in which case a random seed is generated and returned in seed_cur.

@ngxson
Copy link
Collaborator

ngxson commented Oct 8, 2024

Breaking change #9776 : better security control for public deployments

  • /slots endpoint is now disabled by default, start server with --slots to enable it
  • If an API key is set, all endpoints (including /slots and /props) requires a correct API key to access.
    Note: Only /health and /models are always publicly accessible
  • Setting "system_prompt" is removed from /completions endpoint. It is now moved to POST /props (see documentation)

Please note that GET /props is always enabled to avoid breaking the web UI.

@countzero countzero mentioned this issue Oct 10, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ggerganov @slaren @ngxson and others