Collective Knowledge (CK) in a community project to develop open-source tools, platforms and automation recipes that can help researchers and engineers automate their repetitive, tedious and time-consuming tasks to build, run, benchmark and optimize AI, ML and other applications and systems across diverse and continuously changing models, data, software and hardware.
CK consists of several ongoing sub-projects:
-
Collective Mind framework (CM) (~1MB) - a very light-weight Python-based framework with minimal dependencies to help the community implement, share and reuse cross-platform automation recipes to build, benchmark and optimize applications on any platform with any software and hardware. Very simply, it extends the cmake concept with reusable automation recipes and workflows written in plain Python or native OS scripts, accessible via a human readable interface with simple tags, and shareable in public and private repositories in a decentralized way. Furthermore, in comparison with cmake, these automation recipes can not only detect missing code but also download artifacts (models, data sets), preprocess them, build missing dependencies, install them and run the final code on diverse platforms in a unified and automated way. You can read more about the CM concept in this presentation.
-
CM automation recipes for MLOps and DevOps (~6MB) - a small collection of portable, extensible and technology-agnostic automation recipes with a human-friendly interface (aka CM scripts) to unify and automate all the manual steps required to compose, run, benchmark and optimize complex ML/AI applications on diverse platforms with any software and hardware: see online catalog and source code.
-
CM automation recipes to reproduce research projects - a unified CM interface to help researchers and engineers access, prepare and run diverse research projects and make it easier to validate them in the real world across rapidly evolving models, data, software and hardware (see our reproducibility initatives and motivation behind this project).
-
-
Collective Knowledge Playground - an open-source platform to list CM scripts similar to PYPI, aggregate AI/ML Systems benchmarking results with CM workflows, and organize public optimization challenges and reproducibility initiatives to find the most performance and cost-effective AI/ML Systems.
- CK GUI to run modular benchmarks - such benchmarks are composed from CM scripts and can run via a unified CM interface.
2022-2024 MLCommons
- ACM REP'23 keynote about MLCommons CM: [ slides ]
- ACM TechTalk'21 about automating research projects: [ YouTube ] [ slides ]
We plan to rewrite and simplify the CM documentation and tutorials based on user feedback in Q2 2024 - please stay tuned for more details.
Collective Mind workflow automation framework and Collective Knowledge Playground are being developed by the MLCommons Task Force on Automation and Reproducibility as a community effort. Volunteers are very welcome to help with this community project!
CK and CM are community projects based on the feedback from our users and MLCommons members. We would like to thank all collaborators and contributors for their support, fruitful discussions, and useful feedback!