Cosmos-Curate is a powerful video curation system that processes, analyzes, and organizes video content using advanced AI models and distributed computing.

Multi functional app to find duplicates, empty folders, similar images etc.

Python library for building and running distributed data pipelines using Ray

Port of OpenAI's Whisper model in C/C++

Open source project for data preparation of LLM application builders

Scalable data pre processing and curation toolkit for LLMs

A one-stop data processing system to make data higher-quality, juicier, and more digestible for (multimodal) LLMs! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷为大模型提供更高质量、更丰富、更易”消化“的数据！

🔥🔥🔥 [IEEE TCSVT] Latest Papers, Codes and Datasets on Vid-LLMs.

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

A tool to configure, launch and manage your machine learning experiments.

Distributed data engine for Python/SQL designed for the cloud, powered by Rust

The Auron accelerator for distributed computing framework (e.g., Spark) leverages native vectorized execution to accelerate query processing

Developer-friendly, embedded retrieval engine for multimodal AI. Search More; Manage Less.

Transform data, train models, and run SQL with marimo — feels like a next-gen reactive notebook, stored as Git-friendly Python. Deploy as scripts, pipelines, endpoints, and apps. All from an AI-nat…

Curate better data for LLMs

Refine high-quality datasets and visual AI models

Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.

Cross-platform, customizable multimedia/video processing framework. With strong GPU acceleration, heterogeneous design, multi-language support, easy to use, multi-framework compatible and high perf…

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, Du…

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Curated list of datasets and tools for post-training.

#1 Locally hosted web application that allows you to perform various operations on PDF files

LakeSail's computation framework with a mission to unify batch processing, stream processing, and compute-intensive AI workloads.

verl: Volcano Engine Reinforcement Learning for LLMs

Lance Format and Ray

Distilabel is a framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.