DEV Community

#llamacpp

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
llama-server's prompt cache reuses KV computed under a different LoRA scale
homelabpm profile

llama-server's prompt cache reuses KV computed under a different LoRA scale

#llamacpp #llm #selfhosted #lora
Comments
6 min read
Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)
kharlampiiev profile

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

#ai #llm #machinelearning #llamacpp
Comments
6 min read
Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #machinelearning
Comments
8 min read
Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #selfhosted
Comments
7 min read
When Chat Templates Go Wrong
pielounw profile

When Chat Templates Go Wrong

#llm #opensource #ai #llamacpp
Comments
7 min read
Benchmarking Gemma 4 E2B on CPU with llama.cpp: A Practical Local AI Experiment
hardyweb profile

Benchmarking Gemma 4 E2B on CPU with llama.cpp: A Practical Local AI Experiment

#ai #llamacpp #linux #opensource
Comments 4
10 min read
DFlash-2: Benchmarking Z-Lab's Successor to DFlash for Accuracy and Throughput Gains
oooocean66 profile

DFlash-2: Benchmarking Z-Lab's Successor to DFlash for Accuracy and Throughput Gains

#ai #llm #llamacpp #performance
1
Comments 7
14 min read
llama.cpp vs Ollama in 2026: Which Runtime Should You Run?
rosgluk profile
Rost

llama.cpp vs Ollama in 2026: Which Runtime Should You Run?

#llamacpp #ollama #llm #selfhosting
Comments
18 min read
A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x
xbill profile

A 4 GB Laptop GPU vs a 6-Core CPU on Gemma 4, Re-Measured in ABBA Order: 4.1x

#gemma #llamacpp #cuda #benchmarking
10
Comments 2
11 min read
ROCm vs Vulkan for AMD Local LLM Hosting: 2026 Guide
rosgluk profile
Rost

ROCm vs Vulkan for AMD Local LLM Hosting: 2026 Guide

#llm #selfhosting #llamacpp #ollama
Comments 2
20 min read
KV Cache on 16 GB GPUs: Making Long Context Actually Fit
rosgluk profile
Rost

KV Cache on 16 GB GPUs: Making Long Context Actually Fit

#llm #llamacpp #vllm #ollama
Comments 1
22 min read
How to pick --n-cpu-moe in llama.cpp: Qwen3.6 35B-A3B on 12, 16 and 24 GB GPUs
donald_lee_707b59c456e304 profile

How to pick --n-cpu-moe in llama.cpp: Qwen3.6 35B-A3B on 12, 16 and 24 GB GPUs

#llm #llamacpp #localai #gpu
1
Comments 2
3 min read
llama-server's logprobs are placeholders when speculative decoding is on
homelabpm profile

llama-server's logprobs are placeholders when speculative decoding is on

#llamacpp #llm #selfhosted #ai
1
Comments 2
7 min read
Gemma 4 at Over 70 Tokens/s on a 2021 Laptop's 4 GB GPU: The Live Demo, Step by Step
xbill profile

Gemma 4 at Over 70 Tokens/s on a 2021 Laptop's 4 GB GPU: The Live Demo, Step by Step

#gemma #llamacpp #cuda #huggingface
9
Comments 2
12 min read
Inside My llama.cpp Setup: Tuning Qwen 3.8 27B for 512K Context
dmitryame profile

Inside My llama.cpp Setup: Tuning Qwen 3.8 27B for 512K Context

#ai #llm #localllm #llamacpp
Comments 3
7 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.