This page describes the purpose, scope, and high-level structure of the LujiaJin/MiniCPM-o repository. It covers what model families are present, what the two product tracks are, and how the codebase is divided into fine-tuning, deployment, inference, and evaluation subsystems. For detailed information about specific models, see the Model Family page (see 1.1). For fine-tuning procedures, see Fine-tuning. For deployment, see Deployment. For inference, see Inference.
This repository is the official open-source codebase for the MiniCPM-o and MiniCPM-V multimodal large language model (MLLM) families, developed by OpenBMB. It provides:
finetune/)deploy/)web_demos/)eval_mm/)The repository targets on-device and server-side deployment of compact multimodal models that accept image, video, audio, and text inputs and produce text and speech outputs.
Sources: README.md1-36
Two distinct product tracks exist within this repository:
| Track | Latest Model | Parameters | Modalities | Key Feature |
|---|---|---|---|---|
| MiniCPM-V | MiniCPM-V 4.0 | 4B | Vision + Text | On-device deployment; surpasses GPT-4.1-mini on OpenCompass |
| MiniCPM-o | MiniCPM-o 4.5 | 9B | Vision + Audio + Text | Full-duplex multimodal live streaming; approaches Gemini 2.5 Flash |
The MiniCPM-V track focuses on efficient vision-language capability suitable for smartphones and edge devices. The MiniCPM-o track extends this with audio input/output and a full-duplex streaming mechanism that allows simultaneous video/audio input and text/speech output without mutual blocking.
Sources: README.md30-36
Top-level directory layout:
MiniCPM-o/
├── finetune/ # Fine-tuning pipeline (SFT + LoRA)
│ ├── finetune.py # Training entrypoint
│ ├── dataset.py # SupervisedDataset
│ ├── trainer.py # CPMTrainer
│ ├── finetune_ds.sh # DeepSpeed launcher (full-parameter)
│ └── finetune_lora.sh# DeepSpeed launcher (LoRA)
├── deploy/ # Production deployment (Docker Compose)
│ ├── model_server.py # FastAPI backend
│ └── docker-compose.yml
├── web_demos/ # Interactive demo server
│ └── model_server.py # FastAPI + WebSocket + REST
├── eval_mm/ # VQA evaluation
│ └── minicpmv.py # Model wrapper classes
├── chat.py # MiniCPMVChat direct inference dispatcher
└── docs/ # Technical reports and guides
Sources: README.md110-132 docs/llamafactory_train_and_infer.md1-15
Repository subsystems and code entry points:
Sources: README.md110-132 docs/llamafactory_train_and_infer.md14-18
Built end-to-end from four components:
| Component | Role |
|---|---|
Qwen3-8B | LLM backbone |
SigLip2 | Vision encoder |
Whisper-medium | Audio encoder |
CosyVoice2 | Speech decoder |
Key capabilities:
OpenCompass average score (instruct mode): 77.6 across 8 benchmarks.
Sources: README.md136-158
Sources: README.md35
Subsystem map with primary code artifacts:
Sources: README.md110-132 docs/llamafactory_train_and_infer.md1-30
The repository provides integration guides for the following external training and serving frameworks:
| Framework | Supported Models | Use Case |
|---|---|---|
| LLaMA-Factory | MiniCPM-V-4, MiniCPM-o-2.6, MiniCPM-V-2.6 | LoRA SFT, full SFT, web chat inference |
| SWIFT | MiniCPM-V, MiniCPM-o | CLI fine-tuning and inference |
| Xinference | MiniCPM-V variants | Web-based model serving |
| llama.cpp | MiniCPM-V 2.6, MiniCPM-o 2.6, MiniCPM-o 4.5 | CPU inference (GGUF format) |
| Ollama | MiniCPM-o 2.6 | One-click local serving |
| vLLM | MiniCPM-V 2.5+, MiniCPM-o 2.6+ | High-throughput inference |
| SGLang | MiniCPM-V 4.5+ | High-throughput inference |
| FlagOS | MiniCPM-o 4.5 | Multi-chip backend plugin |
Sources: README.md73-77 README.md122-132 docs/llamafactory_train_and_infer.md14-18
| Area | Wiki Page |
|---|---|
| Model family lineage and version history | 1.1 — Model Family |
| MiniCPM-o 4.5 architecture detail | 1.1.1 — MiniCPM-o 4.5 |
| MiniCPM-V 4.0 detail | 1.1.2 — MiniCPM-V 4.0 |
| Fine-tuning pipeline | 2 — Fine-tuning |
| Docker Compose production deployment | 3 — Deployment |
Direct Python inference via chat.py | 4 — Inference |
| VQA evaluation | 5 — Evaluation |
| License and community | 6 — License and Community |
Sources: README.md1-132