Project description

RMTC is a VFX specific framework for simplifying the production and deployment of AI/ML models using well tracked and rights cleared datasets. We wish to develop Python tooling for tracking the training story of a model and connect back to source and rights holders.

AI/ML is continuing to establish itself as an important part of the ongoing efficiency drive to reduce shot costs as well as improve quality. Facilities are active in applying AI but often doing so in a non-centralized manner. Rights and licensing issues, training reproducibility, tooling, datasets, compute and accessibility of model production to artists and TDs are all significant problems.

We are seeing improvements of large-scale video and image generation systems to the point that they are reaching levels of quality that put them in a useful position within a VFX pipeline, but questions remain about the legality of the datasets and the ethics of using models trained on artists data where the rights to inclusion were not specifically granted.

By focusing on the process of formalizing the tracking of datasets, weights and models when training and during inference with the aim of bringing increased rigor around provenance of datasets, we can look to leverage the advancements in AI/ML but in a way that both respects rightsholders and artists intentions as well as ensure facilities meet their regulatory obligations.

This formalization track consists of 3 loosely coupled themes:

1. Track - a Python tracking solution to store the provenance of models and datasets that is training framework agnostic
2. Train - a reference training system that works with the tracking layer to create simple associative models with PyTorch
3. Infer - abstracted DCC inference to track back from inferred result to the model and datasets used, beginning with Image2Image models

Secondary technical goals for this project include standardization of inference model formats, adoption of a reference AI platform, standardized VFX asset tensor formats and conversion, efficient DCC asset to tensor conversion and adoption of safetensors for tensor weights.

Sponsor from TAC

Kimball Thurston

Proposed Project Stage

Sandbox

Please explain how this project is aligned with the mission of the Academy Software Foundation?

AI/ML is changing rapidly and being deployed across VFX facilities in various ways, multiple facilities are independently tackling the same repeated problems around the integration of model production and deployment. RMTC aims to alleviate this redundancy through sharing resources around a singular method for managing data provenance, simplified training and standards for inference.

By collaborating on this project we can also look to tackle shared questions around the role of AI/ML in VFX and set expectations that benefit the industry as a whole - respect for rights and the efforts of the artists involved. RMTC is an opportunity for facilities to show commitment to protecting their creative teams.

RMTC is a project that embodies the spirit of VFX as a creative and collaborative pursuit and could be used to shape the industry narrative around AI/ML in the future.

What is the project’s license for code contributions and methodology for code contributions?

The code will be split into 3 key layers:

• RMTC – Base types and fundamental abstractions
• RMTC Core – A reference implementation of the Track/Train/Infer themes based upon OSS systems
• RMTC Xxxx - Facility specific implementations

Apache-2.0 will cover the API & reference implementation with proprietary licenses covering the facility specific portions.

What tool or platform is utilized for source control (GitHub, etc.), and what is the location

(e.g., URL)?

GitHub – the repository is to be set up upon confirmation of the project with ASWF.

What are the external dependencies of the project, and what are the licenses of those dependencies?

We propose as part of this project an extension to the VFX Platform to include AI/ML frameworks, specifically the following:

Torch – for training with the reference implementation

• PyTorch
• OpenCV
• TorchVision
• CUDA

ONNX – for DCC based inference and to encourage ONNX inference standardization in VFX

• ONNX
• ONNX Script
• ONNX Runtime

What roles does the project have (e.g., maintainers, committers?) Who are the current

core committers of the project, or which can a list of committers be found?

The project is currently a single engineer – John McCarten. Kimball Thurston is the current chief stakeholder. The project is expanding to 3 engineers as and when internal resourcing allows.

What mailing lists are currently used by the project?

There is no current mailing list.

What tool or platform is leveraged by the project for issue tracking?

The project is coarsely planned with MS Project and execution tracking will be via an internal JIRA instance. We will consider moving to GitHub roadmap and reporting systems as we move public.

Does the project have an OpenSSF Best Practices Badge? Do you foresee any

challenges in obtaining one?

The project is currently internal so does not have an OpenSSF badge. We can start with the baseline badge then to obtain higher grades, we would need to secure talent to cover secure software design.

What is the project’s website? Is there a wiki?

The project is currently internal to the facility and has presence on the internal Confluence instance.

What social media accounts are used by the project?

This is currently an internal project so no public social media – there is an expectation of creating social media presence once accepted by the ASWF as an active project.

What is the project’s release methodology and cadence?

We are following a waterfall/agile project structure – coarse planning through milestone deliveries that describe high level functionality and agile sprints to deliver that milestone. We anticipate flexibility at the detail level with constant priority adjustment via stakeholder alignment during sprint planning.

The execution of those milestones is planned via an agile/stakeholder program running as 3 week sprints, with milestone delivery resulting in quarterly release after a 3 week code freeze.

Each milestone will be delivered as a live demo to stakeholders before adaption into the production pipeline.

Are any trademarks, registered or unregistered, leveraged by the project? Have any

trademark registrations been filed by the project or any third party anywhere in the world?

No trademarks have been registered or leveraged – the name itself is unique to the region and unlikely to be in use.

What existing maintainers and contributors does the project have? Are there

organizations involved that are committing resources to the project; if so
please describe.

The project currently has a fulltime engineer and part time program manager. There are additional staff planned to onboard from internal teams in the coming months.