Optuna: a Black-Box Optimization Framework

Transcript

1. Optuna: a Black-Box Optimization Framework Preferred Networks, Inc. Hideaki Imamura

2. 2 Hideaki Imamura • Researcher at Preferred Networks, Inc •

   Research on the theory and applications of black-box optimization • Led Optuna v3.0 and v3.6 • GitHub: @HideakiImamura

3. Optuna: a Black-Box Optimization Framework • Hyperparameters (e.g., learning rate)

   control behavior of models and algorithms • Optuna finds the best hyperparameters via trial-and-error 3 ML Model ML Model ML Model

4. 4 Optuna Users in Numbers 10k GitHub stars 16k repositories

   3M monthly downloads Used by

5. 5 • Over 20 maintainers ◦ From various companies and

   universities • Over 240 developers • Working with OSS communities & 140+ more contributors Optuna Contributors in Numbers

6. 6 • Optuna has been globally recognized and seen continuous

   growth by adapting to user demands • Collaboration with ML-related OSS and knowledge exchange among users and developers has extended Optuna's reach beyond AI competitions to practical applications • The latest initiative, OptunaHub, aims to be a platform that disseminates rich, user-generated contents across its rapidly expanding communities Outline: Success and Future of Optuna

7. 7 The Origin of Optuna How did the development start?

8. 8 Development Started as an Internal Tool • Initially designed

   for deep learning hyperparameter optimization • Contributed to the 2nd place in Google AI Open Images Challenge ◦ [1809.00778] PFDet: 2nd Place Solution to Open Images Challenge 2018 Object Detection Track Ref: https://www.kaggle.com/c/google-ai-open-images-object-detection-track

9. 9 Optuna Was Released on GitHub as OSS Presentations Documentations

   Paper (KDD2019) Ref: https://arxiv.org/abs/1907.10902 Ref: https://optuna.org/

10. 10 Why Did We Release Optuna as OSS? Quote from

    Takuya Akiba, one of the Optuna launch members: "We … place great importance on presenting our study results to contribute to the community. This … will eventually accelerate the advances of the technology … if we can inspire other professionals in the world… This is why we are very active in … releasing software as an OSS.

11. 11 is one of the most important applications of Optuna

    Early Adopters: ML Competitions Integration with LightGBM Popular Notebook by Grandmaster

12. 12 Our User Community Has Expanded Integrations of many machine

    learning libraries Medium

13. 13 Industrial Applications of Optuna How have our products been

    used for real-world problems?

14. 14 Optuna Extends Preferred Networks (PFN) Businesses Materials Science Autonomous

    Mobile Robot Multimodal Foundation Models

15. 15 Robotics • PFN is conducting R&D on robotics and

    selling toC products • Optuna contributed to accurate robot vision with low latency Autonomous Mobile Robot: kachaka™ (Now on sale in Japan) Company: https://www.pfrobotics.jp/ Kachaka: https://kachaka.life/ Video: https://youtu.be/u_i2X43dBGw

16. 16 Robot Needs Semantic Segmentation • Semantic segmentation is essential

    for autonomous robot • Neural architecture is important for high accuracy and low latency

17. 17 Neural Architecture Search (NAS) by Optuna Lower latency but

    lower mIoU Higher mIoU but higher latency mIoU Latency (ms) • Introduced multi-objective optimization into Optuna • Got better trade-offs of the mIoU and latency compared to the baseline model • Introduced constrained optimization to limit to the acceptable latency MobileNetV2 (Baseline)

18. 18 Materials Science • Matlantis™ supports large-scale materials discovery by

    simulating the behavior of various materials at an atomic level • Used for energy calculations of material structures Matlantis™ Material Structure (e.g., Ti₂O) Energy 10K Times Faster than conventional methods E( ) E( )

19. 19 Nature Solves an "Annealing" Problem In general, the materials

    that actually appear in the world have the structure with the lowest energy. Same chemical formula Ti 2 O Has the lowest potential energy Stable structure in nature Has relatively higher potential energy Unstable structure in nature

20. 20 Crystal Structure Prediction (CSP) … Done! Energy: -3.0 First,

    I want to try this: Structure: Trial 1 … Done! Energy: -1.7 Then, how about this?: Structure: Trial 2 … Done! Energy: -3.3 So, I want to try this: Structure: Trial 3 • CSP is formulated as a black-box optimization problem • Input is the structure suggested by Optuna • Output is the energy calculated by Matlantis

21. 21 Problem: Crystal Structure Prediction (CSP) ~ Part I ~

    Ti x O 1-x , x=2/3 → Ti ⅔ O ⅓ = Ti 2 O Same composition (Ti₂O) but different structures yield different energies Better Composition Ratio x

22. 22 Problem: Crystal Structure Prediction (CSP) ~ Part II ~

    Better Existing method: Optimize structure for each composition x Composition Ratio x

23. 23 Problem: Crystal Structure Prediction (CSP) ~ Part III ~

    Better Objective: Maximize the blue shaded area Optimal energies for each structure Composition Ratio x

24. 24 Problem: Crystal Structure Prediction (CSP) ~ Part IV ~

    Better Proposed method: Jointly optimize over all compositions in the multi-objective optimization manner! Composition Ratio x

25. 25 Found New Structures by Our Method Al₂MnCu • Found

    new stable structures for multiple elements at various composition ratios • Achieved good sample efficiency using correlation between structures of different composition ratios New Known Ti₂O Known Ca 3 P 2 New Known New

26. 26 Other Applications Some real-world application examples have been reported

    by the community. Automated Taxi Dispatch （By Mobility Technololgies at April 2020） Blog Post JVM Parameter Tuning （By UL Systems at June 2021） Ref: https://wiki.openjdk.org/display/zgc/Main Ref: https://www.slideshare.net/slideshow/jvmoptuna-optuna-meetup-1/249496426 Obtained Parameter Default Parameter Better

27. 27 The Future of Optuna How will the ecosystem be?

28. 28 Diverse Needs for Optuna from Real-world Applications Extended problem

    setup Multi-objective optimization Constrained optimization Batch optimization Multi-fidelity optimization etc… Cutting edge algorithm Algorithms based on the recent academic papers Domain specific method Each real-world application often requires its own unique considerations Incorporating all needs into Optuna is difficult from a software design perspective Benefit the community by sharing assets gained through real-world applications

29. 29 OptunaHub: a Feature-Sharing Platform • Contributors (researchers and developers)

    can easily register their own algorithms to OptunaHub • Users can use those features in Optuna manners

30. 30 OptunaHub is Growing with Contributors Materials Science resnant@Toyota Evolutionary

    Computation nomuramasahir0@CyberAgent Hyperparameter Optimization [email protected] From the BBO-Rietveld paper under a Creative Commons Attribution 4.0 International License PFNs4BO CatCMA BBORietVeld From https://arxiv.org/abs/2305.17535 From https://arxiv.org/abs/2405.09962

31. 31 Submit a pull request according to the tutorial!!! How

    to Register Your Algorithm into OptunaHub The top page of OptunaHub The detail page of a feature

32. 32 How to Use OptunaHub $ pip install optunahub Step1:

    Installation Step2: Load Module and Use! More API details: https://optuna.github.io/optunahub/ Visit hub.optuna.org for the list of all features

33. 33 Wrap up • Optuna: a globally recognized black-box optimization

    framework ◦ 3M+ downloads/month, 10K+ GitHub stars, 16K+ used in GitHub repos • Continuous growth by adapting to user demands ◦ From Kaggler to ML Practitioners ◦ Integrations of ML libraries such as LightGBM and PyTorch • Industrial Applications ◦ Robotics & Materials Science • Further expansion of Optuna Community ◦ New feature-sharing platform OptunaHub $ pip install optuna optunahub

34. 34 We are looking forward to collaborations with you! Visit

    optuna.org for more information Contact at the GitHub issue/discussions

35. Making the real world computable