Tsukaima is a tool to call local large language models (LLMs) using the existing OpenAI ChatGPT clients. Currently, the following models are supported.

• rinna LLM
• line-corporation/japanese-large-lm-3.6b-instruction-sft
• elyza/ELYZA-japanese-Llama-2-7b-instruct

$ python3 -m venv myvenv
$ source myvenv/bin/activate
(myvenv) $ pip install -U git+https://github.com/shirayu/tsukaima.git
(myvenv) $ wget https://raw.githubusercontent.com/shirayu/tsukaima/main/examples_config/rinna.json -O rinna.json
(myvenv) $ tsukaima --host 0.0.0.0 --port 6006 --config ./rinna.json

Set API endpoint to the address (Eg: http://0.0.0.0:6006/v1/chat/completions) to use ChatGPT clients such as BetterChatGPT.

Check other config examples.

• Messages whose role is system will be ignored

Please read tsukaima.schema.schema

• You may need install NCCL
  • sudo apt install libnccl-dev libnccl2

• https://gist.github.com/advanceboy/b9143aa9de23a6f9a60a07a862e0b4a8
• https://note.com/hamachi_jp/n/n70b1e48c09a0#816e2412-a3b6-4319-b056-3a7a1bcf7638
• https://note.com/hamachi_jp/n/n8e1cbf3314be
• https://github.com/lm-sys/FastChat
• https://gist.github.com/advanceboy/b9143aa9de23a6f9a60a07a862e0b4a8