Markio

API-first document parsing platform built with FastAPI, Docling, and MinerU

English | 中文

Markio converts documents and web content into Markdown or structured text through four delivery surfaces:

• FastAPI sync parsing endpoints under /v1/parse_*
• FastAPI async task queue endpoints under /v1/tasks/*
• Local Python SDK and CLI
• Vue 3 console served by FastAPI at /console

The project is currently alpha (0.1.5). It is suitable for internal environments, staged rollouts, and integration work, but it is not positioned as a fully hardened GA platform yet.

• Primary web UI: Vue console at /console
• Optional auxiliary UI: Gradio preview/demo workflow
• Auth model: every /v1/* route requires Authorization: Bearer <JWT>
• Task backend: in-memory by default, Redis optional
• URL parsing: local parser, SDK local mode, and remote /v1/parse_url now share the same URL safety model

flowchart LR
    A["Clients (REST / SDK / CLI / Console)"] --> B["FastAPI App"]
    B --> C["Sync Parse Routers"]
    B --> D["Async Task Router"]
    B --> E["Static Console Mount (/console)"]
    C --> F["Parser Registry + Request Guards"]
    F --> G["Docling / MinerU / URL / Bio Parsers"]
    D --> H["Task Manager (memory or Redis)"]
    H --> G
    H --> I["Redis Task Store / Cache (optional)"]

• Python 3.11+
• uv recommended
• Node.js 18+ for frontend development and console builds
• Optional: Docker + Docker Compose
• Optional: LibreOffice for .doc and .ppt
• Optional: Redis when using TASK_QUEUE_BACKEND=redis

git clone https://github.com/Tendo33/markio.git
cd markio

uv sync
uv pip install -e .

cp .env.example .env
python markio/main.py

Required before the app can start:

• set AUTH_JWT_SECRET in .env

Open:

• API docs: http://localhost:8000/docs
• Health: http://localhost:8000/healthz

The backend serves the Vue SPA only when the built frontend assets exist in markio/webapp.

cd frontend
pnpm install
pnpm run build
cd ..

Then open:

• Console: http://localhost:8000/console

If the build output is missing, /console intentionally returns a helper fallback page instead of a broken SPA shell.

export AUTH_JWT_SECRET="<strong-random-secret>"
export REDIS_PASSWORD="<redis-password>"
docker compose up -d

Compose keeps Redis internal-only and aligns with same-origin hosting: API at /v1/*, console at /console.

Project facts for AI-assisted development live in .trellis/spec/. Start with .trellis/spec/README.md, then read the backend or frontend spec index and .trellis/spec/shared/verification.md before changing routes, parsers, tasks, SDK/CLI behavior, console UI, or deployment assumptions.

curl -X POST "http://localhost:8000/v1/parse_file" \
  -H "Authorization: Bearer <YOUR_JWT>" \
  -F "file=@./sample.docx"

curl -X POST "http://localhost:8000/v1/parse_url?url=https://example.com" \
  -H "Authorization: Bearer <YOUR_JWT>"

curl -X POST "http://localhost:8000/v1/tasks/submit" \
  -H "Authorization: Bearer <YOUR_JWT>" \
  -F "file=@./sample.pdf" \
  -F "parse_method=auto" \
  -F "lang=ch" \
  -F "priority=5"

curl -H "Authorization: Bearer <YOUR_JWT>" \
  "http://localhost:8000/v1/tasks?page=1&page_size=20"

curl -H "Authorization: Bearer <YOUR_JWT>" \
  "http://localhost:8000/v1/tasks/dashboard"

curl -H "Authorization: Bearer <ADMIN_JWT>" \
  "http://localhost:8000/v1/tasks/queue"

Editable install exposes the markio CLI.

markio pdf ./sample.pdf --method auto
markio docx ./sample.docx --save
markio url https://example.com

Remote mode:

markio --api-base-url http://localhost:8000 --token <YOUR_JWT> pdf ./sample.pdf --save

Python SDK:

import asyncio
from markio.sdk.markio_sdk import MarkioSDK

async def main():
    sdk = MarkioSDK(output_dir="outputs")
    result = await sdk.parse_pdf("sample.pdf", parse_method="auto")
    print(result["content"][:500])

asyncio.run(main())

Remote SDK mode:

sdk = MarkioSDK(
    output_dir="outputs",
    api_base_url="http://localhost:8000",
    token="<YOUR_JWT>",
)

• All /v1/* routes require JWT auth
• role=admin is required for queue pause/resume routes
• /v1/tasks/dashboard is owner-scoped; users only see their own task stats and recent items
• The console keeps the token in frontend-managed browser state and persists it via localStorage; expired or malformed browser tokens are treated as unavailable until replaced

• /v1/mcp/* and legacy /mcp/* use the standard non-2xx FastAPI error envelope for validation and parser failures
• legacy /mcp/* routes remain deprecated and still emit deprecation headers

URL fetching is constrained by one shared implementation in markio/parsers/url_parser.py:

• only http and https
• optional domain allowlist via URL_ALLOWED_DOMAINS
• private, loopback, link-local, multicast, reserved, and unspecified addresses blocked by default
• timeout, payload-size, and redirect limits
• redirect targets revalidated before follow
• validated IPs pinned into the actual connection layer for direct fetch mode

Relevant environment variables:

• URL_FETCH_MODE
• URL_PROXY_BASE
• URL_REQUEST_TIMEOUT_SECONDS
• URL_MAX_RESPONSE_BYTES
• URL_BLOCK_PRIVATE_NETWORKS
• URL_ALLOWED_DOMAINS
• URL_MAX_REDIRECTS

Core configuration comes from .env and markio/settings/config_model.py.

Redis-specific details: docs/REDIS_INTEGRATION.md

• CLI guide: docs/cli_usage.md
• SDK guide: docs/sdk_usage.md
• Console guide: docs/console_frontend.md
• Biological parsing guide: docs/biological_data_parsing.md
• Redis guide: docs/REDIS_INTEGRATION.md
• Test guide: tests/README.md

Primary commands:

uv run pytest
uv run pytest -m live
cd frontend && pnpm run build

The repository currently uses pytest as the source of truth. Legacy helper scripts under tests/ still exist, but direct pytest runs are the preferred path.

markio/
├── markio/         # FastAPI app, parsers, routers, SDK, settings
├── frontend/       # Vue 3 + Vite console source
├── docs/           # User and operator documentation
├── tests/          # Pytest suite and test fixtures
├── data/           # Runtime task state/uploads
├── logs/           # Runtime logs
└── outputs/        # Saved parse outputs

• The project is still marked alpha, not GA
• Console auth remains token-based on the frontend side for now
• Queue pause/resume and global queue health are admin-only; dashboard remains owner-scoped
• There is no first-class CLI or MarkioSDK wrapper for FASTA and GenBank yet; use the REST endpoints or parser modules directly
• The fallback page at /console is intentional and only exists to signal missing build assets