Experimental Scala 3 bindings for llama.cpp using Slinc.
Add llm4s to your build.sbt:
libraryDependencies += "com.donderom" %% "llm4s" % "0.22.0-b10273"For JDK 17 add .jvmopts file in the project root:
--add-modules=jdk.incubator.foreign
--enable-native-access=ALL-UNNAMED
- Scala: 3.3.0
- JDK: 17 or 19
llama.cpp: The version suffix refers to the latest supportedllama.cpprelease (e.g. version0.22.0-b10273means that it supports the b10273 release). The newer releases are usually supported as well, provided there are no API changes.
Older versions
| llm4s | Scala | JDK | llama.cpp (commit hash) |
|---|---|---|---|
| 0.11+ | 3.3.0 | 17, 19 | 229ffff (May 8, 2024) |
| 0.10+ | 3.3.0 | 17, 19 | 49e7cb5 (Jul 31, 2023) |
| 0.6+ | 3.3.0-RC3 | --- | 49e7cb5 (Jul 31, 2023) |
| 0.4+ | 3.3.0-RC3 | --- | 70d26ac (Jul 23, 2023) |
| 0.3+ | 3.3.0-RC3 | --- | a6803ca (Jul 14, 2023) |
| 0.1+ | 3.3.0-RC3 | 17, 19 | 447ccbe (Jun 25, 2023) |
import java.nio.file.Paths
import com.donderom.llm4s.*
// Path to the llama.cpp shared library
System.load("./build/bin/libllama.dylib")
// Path to the model supported by llama.cpp
val model = Paths.get("Llama-3.2-3B-Instruct-Q6_K.gguf")
val prompt = "What is LLM?"val llm = Llm(model)
// To print generation as it goes
llm(prompt).foreach: stream =>
stream.foreach: token =>
print(token)
// Or build a string
llm(prompt).foreach(stream => println(stream.mkString))
llm.close()val llm = Llm(model)
llm.embeddings(prompt).foreach: embeddings =>
embeddings.foreach: embd =>
print(embd)
print(' ')
llm.close()Self-contained Scala CLI examples
Basic Llama 3 model
Run.scala:
//> using scala 3.3.0
//> using jvm adoptium:17
//> using java-opt --add-modules=jdk.incubator.foreign
//> using java-opt --enable-native-access=ALL-UNNAMED
//> using dep com.donderom::llm4s:0.22.0-b10273
import com.donderom.llm4s.Llm
import java.nio.file.Paths
import scala.util.Using
object Main extends App:
System.load("./build/bin/libllama.dylib")
// Path to the downloaded model (models are not downloaded automatically)
val model = Paths.get("Llama-3.2-3B-Instruct-Q6_K.gguf")
val prompt = "What is LLM?"
Using(Llm(model)): llm => // llm : com.donderom.llm4s.Llm
llm(prompt).foreach: stream => // stream : LazyList[String]
stream.foreach: token => // token : String
print(token)scala-cli Run.scalaConfigured gpt-oss model
Run.scala:
//> using scala 3.3.0
//> using jvm adoptium:17
//> using java-opt --add-modules=jdk.incubator.foreign
//> using java-opt --enable-native-access=ALL-UNNAMED
//> using dep com.donderom::llm4s:0.22.0-b10273
import com.donderom.llm4s.{ContextParams, FlashAttention, Llm, LlmParams}
import java.nio.file.Paths
import scala.util.Using
object Main extends App:
System.load("./build/bin/libllama.dylib")
// Path to the downloaded model (models are not downloaded automatically)
val model = Paths.get("gpt-oss-20b-mxfp4.gguf")
val prompt = "What is LLM?"
// Use Flash attention and context size provided by the model
val params = LlmParams(context = ContextParams(flashAttention = FlashAttention.On))
Using(Llm(model)): llm => // llm : com.donderom.llm4s.Llm
llm(prompt, params).foreach: stream => // stream : LazyList[String]
stream.foreach: token => // token : String
print(token)scala-cli Run.scalaConfigured Qwen3.8 model using recommended settings
Run.scala:
//> using scala 3.3.0
//> using jvm adoptium:17
//> using java-opt --add-modules=jdk.incubator.foreign
//> using java-opt --enable-native-access=ALL-UNNAMED
//> using dep com.donderom::llm4s:0.22.0-b10273
import com.donderom.llm4s.{ContextParams, FlashAttention, Llm, LlmParams, Sampling}
import java.nio.file.Paths
import scala.util.Using
object Main extends App:
System.load("./build/bin/libllama.dylib")
// Path to the downloaded model (models are not downloaded automatically)
val model = Paths.get("Qwen3.8-27B-Q4_1.gguf")
val prompt = "What is LLM?"
// Use Flash attention and sampling parameters for instruct mode
val params = LlmParams(
context = ContextParams(flashAttention = FlashAttention.On),
sampling = Sampling.Dist(temp = 0.7, topK = Some(20), minP = None)
)
Using(Llm(model)): llm => // llm : com.donderom.llm4s.Llm
llm(prompt, params).foreach: stream => // stream : LazyList[String]
stream.foreach: token => // token : String
print(token)scala-cli Run.scalaApart from the model path, the examples above differ in what configuration params passed to the model.
There are three entry points for configuring the model:
- ModelParams are passed once when an Llm is instantiated
- LlmParams containing most of parameters which are optionally passed to every generation along with the prompt
- EmbeddingParams are used for generating embeddings