DataReady

Multimodal Data Preparation for AI
Parse · Deduplicate · Denoise · Evaluate · Synthesize

Features  ·  Quick Start  ·  Datasets  ·  Documentation  ·  简体中文

DataReady brings together five independent, open-source workflow engines for preparing, improving, evaluating, and generating data for AI. It helps data engineers and researchers turn documents, media, and structured records into usable datasets for model development and downstream applications.

Each toolkit is an independent Python project with its own command-line entry point. YAML workflows define inputs, outputs, processing steps, and operator parameters; custom Python operators extend the built-in capabilities. Runs produce structured results, metrics, and reports, with checkpoint and resume support.

Use one toolkit for a focused task or connect several through files, mapping each toolkit's output fields to the next toolkit's input configuration.

MultimodalParsing converts PDF, Word, Excel, HTML, image, and audio inputs into structured JSONL artifacts. Its parsing workflows bring together content extraction, OCR or ASR integration, Markdown reconstruction, chunking, and quality assessment. Output records retain source information, extracted content, metrics, and issues, making them suitable for subsequent cleaning, indexing, and dataset preparation.

DataDeduplication identifies exact and near-duplicate text, image, and audio samples. Text processing includes normalization, hashing, SimHash, and MinHash LSH; image processing includes file and perceptual hashing and structural similarity; audio processing includes file and PCM hashing, fingerprints, and acoustic similarity. Optional embedding, reranking, and ASR stages extend these workflows. Results include duplicate groups, representative samples, kept and removed records, review candidates, and decision reports.

DataDenoising cleans text, images, image-text pairs, and video, with automatic routing for mixed inputs. Text operations cover normalization, markup cleanup, repetition filtering, and sensitive information masking. Image and video workflows assess decoding, resolution, blur, exposure, noise, and other quality signals; image-text workflows check consistency between modalities. Configurable quality decisions route samples into keep, drop, or manual-review outputs.

QualityEvaluation evaluates text, image, and audio datasets and reports sample-level issues alongside dataset-level summaries. Checks cover field completeness, text length and punctuation, duplicate content, image integrity and annotations, and audio format, sample rate, clipping, noise, and echo. The toolkit produces quality scores, issue distributions, mappings to national-standard quality dimensions, Markdown reports, operator traces, and error queues to support review and improvement.

DataSynthesis generates new samples from seed data, prompts, and schemas. Built-in workflows cover text generation, text and table question answering, synonym replacement, question augmentation, structured-record generation, image generation, and multimodal synthesis. Internal validation, diversity checks, quality gates, and retry logic help separate accepted samples from filtered or failed results. Generated data, metrics, workflow copies, and checkpoints provide a traceable record of each run.

External model setup: OCR, ASR, embedding, vision, and generative-model capabilities require the corresponding backend and credentials. Consult the toolkit guides for integrations and input/output requirements.

Raw data  → Parse      → Deduplicate → Denoise → Evaluate → Usable datasets
Seed data → Synthesize → Deduplicate / Denoise / Evaluate → Expanded datasets

Choose the steps and order for your data and intended use. Toolkits exchange files; each workflow defines its own input and output format.

Requirements: Python >=3.10; uv is recommended. Video denoising also requires ffmpeg / ffprobe.

Each toolkit has its own environment. For the examples below, enter the relevant toolkit directory before using uv run.

Start with structured data synthesis without an external model service. Run from the repository root:

cd DataSynthesis
uv sync
uv run synthesis validate -c workflows/structured_synthesis.yaml
uv run synthesis run -c workflows/structured_synthesis.yaml
uv run synthesis report -t runs/structured_synthesis

Results are written to DataSynthesis/runs/structured_synthesis/. See the DataSynthesis guide for input, output, and report details.

Explore the other entry points from their respective toolkit directories:

We also release five WutongWanxiang datasets on the Huanxin community(https://aihuanxin.cn/#/). They cover cross-cultural language, Chinese cultural imagery, model safety, industry knowledge, and technology-domain speech. Counts and descriptions follow the publishers’ introductions; use each dataset page for details and access.

Expand for content and preparation details:

Dataset and software licenses are separate. These datasets are distributed outside this repository; the linked pages specify CC BY-NC 4.0, while the software uses MIT. Map downloaded text, image, and metadata fields to the selected toolkit’s input configuration.

Workflow configuration references are currently in Chinese.

This project is licensed under the MIT License. See LICENSE.

Copyright © 2026 China Mobile Information Technology

Back to top