High-speed Large Language Model Serving for Local Deployment
-
Updated
Jan 24, 2026 - C++
#
local-inference
Here are 53 public repositories matching this topic...
[ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration
-
Updated
Nov 18, 2024 - Python
Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant
desktop-app rust productivity machine-learning opensource privacy cross-platform offline chatbot svelte artificial-intelligence candle conversational-ai tauri ai-chat llm local-inference tauri-rust local-llm qwen
-
Updated
Mar 24, 2026 - Rust
Mano-P: Open-source GUI-VLA agent for edge devices. #1 on OSWorld (specialized, 58.2%). Runs locally on Apple M4 Mac mini/MacBook — no data leaves your device.Mano-P 是一个开源 GUI-VLA 项目，支持在 Mac mini/MacBook 上或通过算力棒本地运行推理，实现纯视觉驱动的跨平台 GUI 自动化操作。数据完全本地处理，支持复杂多步骤任务规划与执行。
desktop-automation mano gui-automation edge-computing on-device-ai local-inference vision-language-action multimodal-ai gui-grounding osworld computer-use-agents visual-language-model mano-p
-
Updated
Mar 26, 2026
On-device AI for iOS & Android
embeddings gemini http-server openai document-processing rag edge-ai on-device-ai local-inference anthropic llamacpp llama-cpp local-llm gguf multimodal-ai
-
Updated
Mar 29, 2026 - TypeScript
Notolog Markdown Editor
python markdown qt emacs markdown-editor onnx python-ai on-device-ai ai-assistant pyside6 python-qt local-inference llama-cpp local-llm qwen llama-cpp-python gguf phi-4 privacy-first-ai
-
Updated
Feb 7, 2026 - Python
A fully browser-native RAG application for document Q&A, powered by Rust and WebAssembly with local vector search, embeddings, and in-browser LLM inference.
-
Updated
Mar 24, 2026 - JavaScript
Tool for test diferents large language models without code.
api-rest chat-application flask-api inference-api large-language-models llm local-inference llamacpp llm-inference llama2 llama-cpp-python llama2-7b mixtral-8x7b
-
Updated
Oct 18, 2025 - Python
Local AI music generator with smart lyrics: Gradio web UI for HeartMuLa + Ollama/OpenAI, tags, history, and high-fidelity audio.
music gui webui music-generation gradio ai-music lyrics-generator ai-lyrics musicgeneration aimusic local-inference suno-ai musicgen local-ai ollama ai-music-generation suno heartmula heartcodec heartmula-gui
-
Updated
Mar 9, 2026 - Python
Desktop AI tutoring app with local inference using Ollama for privacy-focused education.
electron javascript desktop-app css education privacy typescript ai offline nextjs edtech desktop-application llama tutoring privacy-focused local-inference ollama ai-tutor offline-application
-
Updated
Mar 1, 2026 - TypeScript
LLM chatbot example using OpenVINO with RAG (Retrieval Augmented Generation).
natural-language-processing offline chatbot intel edge-computing rag openvino huggingface edge-inference cloud-free llm local-inference langchain dolly2 retrieval-augmented-generation llama2 neural-chat
-
Updated
Jan 25, 2024 - Python
EN: An overfitted SD prompt engine with severe "aesthetic snobbery," forcibly transforming mundane ideas into professional-grade physical rendering instructions. CN: 一个具备“审美洁癖”的过拟合提示词引擎，强行将平庸构思纠偏为具备极致物理质感的工业级渲染指令。
native cuda self-hosted python3 micro aesthetic industrial-automation echo-server lora gradio fine-tuning ai-art stable-diffusion local-inference prompt-generation qwen local-ai-agents blackwell-gpu anti-hallucination deterministic-ai
-
Updated
Jan 19, 2026 - Python
A lightweight CUDA-based local inference platform built around Z-Image Turbo by Tongyi
-
Updated
Mar 27, 2026 - Python
Llama.cpp but for kindles
-
Updated
Mar 8, 2026
Lightweight 6GB VRAM Gradio web app with auto-installer for running AuraFlow locally — no cloud, no clutter.
python open-source image-generation webui gradio text-to-image stable-diffusion diffusers local-inference generative-ai ai-image-generator auraflow low-vram
-
Updated
Jun 7, 2025 - Python
Local embeddings server for Apple Silicon using MLX, providing OpenAI-compatible API endpoints
machine-learning privacy embeddings m2 m3 mlx m1 text-embeddings vector-embeddings apple-silicon openai-api local-inference qwen litellm openai-compatible
-
Updated
Feb 17, 2026 - Shell
Edge Agent Lab is an Android testing platform for evaluating small language model (SLM) agents directly on mobile devices.
android-development gemma litert edge-ai mediapipe local-inference function-calling agentic-ai koog litert-lm
-
Updated
Mar 29, 2026 - Kotlin
Privacy‑first, real‑time speech‑to‑text dictation. 100% local inference in Rust; hotkey to dictate anywhere (macOS, Linux, Windows).
-
Updated
Mar 18, 2026 - Rust
Verify claims using AI agents that debate using scraped evidence and local language models.
python scraping beautifulsoup autonomous-agents local-inference qwen lm-studio langgraph private-llm agentic-ai claim-verification phi-4-mini evidence-based-ai
-
Updated
Jun 1, 2025 - Python
Windows-based, high-performance, full stack, local LLM chat application written in C# .NET 10 using the Lethe AI library.
-
Updated
Mar 19, 2026 - C#
Improve this page
Add a description, image, and links to the local-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the local-inference topic, visit your repo's landing page and select "manage topics."