ml/data
ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
lakeFS - Data version control for your data lake | Git for data
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Examples of using Neptune to keep track of your experiments (maintenance only).
A kubernetes based framework for hassle free handling of datasets
SeaweedFS is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek, cloud tiering. Filer supports Cloud Drive, xDC replica…
JuiceFS is a distributed POSIX file system built on top of Redis and S3.
For recording and retrieving metadata associated with ML developer and data scientist workflows.
PyTorch extensions for high performance and large scale training.
An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
Alluxio, data orchestration for analytics and machine learning in the cloud
Spark RAPIDS plugin - accelerate Apache Spark with GPUs
The open source developer platform to build AI agents and models with confidence. Enhance your AI applications with end-to-end tracking, observability, and evaluations, all in one integrated platform.
Distributed AI Model Training and Fine-Tuning on Kubernetes
Resource scheduling and cluster management for AI
NVIDIA's launch, startup, and logging scripts used by our MLPerf Training and HPC submissions
A latent text-to-image diffusion model
The Python Risk Identification Tool for generative AI (PyRIT) is an open source framework built to empower security professionals and engineers to proactively identify risks in generative AI systems.
Efficient vision foundation models for high-resolution generation and perception.
[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
VILA is a family of state-of-the-art vision language models (VLMs) for diverse multimodal AI tasks across the edge, data center, and cloud.
[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.
Together Mixture-Of-Agents (MoA) – 65.1% on AlpacaEval with OSS models