Lightning-Hydra-Template

This is an adaptation of Laos_Viet Machine Translation Project to Pytorch Lightning Hydra, progress is still being made.

To run training, simply run train.py file in src folder, all configuration is customized in configs folder.

To run evaluation, simply run eval.py, if you want to run inference only, comment trainer.eval and run trainer.predict instead in the same file !!

The main process is just calling other process, the sub_process is defined in the configs and instantiated by hydra utilization.

The process is still faulty, on progress of resolving to seamless training

A pre-configured vocab is a must, import our vocab or enumerate your own vocab for convenience, each vocab's header should consist 3 components: no. of words, embedding dimension and number of special keywords.

Embedding can also be read with vocab and is stored inside Vocab class.You can just simply run download.sh to import the vocab :D

Why you might want to use it:

✅ Save on boilerplate
Easily add new models, datasets, tasks, experiments, and train on different accelerators, like multi-GPU, TPU or SLURM clusters.

✅ Education
Thoroughly commented. You can use this repo as a learning resource.

✅ Reusability
Collection of useful MLOps tools, configs, and code snippets. You can use this repo as a reference for various utilities.

Why you might not want to use it:

❌ Things break from time to time

I hate this

Lightning and Hydra are still evolving and integrate many libraries, which means sometimes things break. For the list of currently known problems visit this page.

❌ Not adjusted for data engineering

Everytime, new datamodule, sad thing.

Template is not really adjusted for building data pipelines that depend on each other. It's more efficient to use it for model prototyping on ready-to-use data.

❌ Overfitted to simple use case

Very over-fitted, only a transformer can cause so much pain

The configuration setup is built with simple lightning training in mind. You might need to put some effort to adjust it for different use cases, e.g. lightning fabric.

❌ Might not support your workflow

Sad

For example, you can't resume hydra-based multirun or hyperparameter search.

Note: Keep in mind this is unofficial community project.

PyTorch Lightning - a lightweight PyTorch wrapper for high-performance AI research. Think of it as a framework for organizing your PyTorch code.

Hydra - a framework for elegantly configuring complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.

• Rapid Experimentation: thanks to hydra command line superpowers
• Minimal Boilerplate: thanks to automating pipelines with config instantiation
• Main Configs: allow you to specify default training configuration
• Experiment Configs: allow you to override chosen hyperparameters and version control experiments
• Workflow: comes down to 4 simple steps
• Experiment Tracking: Tensorboard, W&B, Neptune, Comet, MLFlow and CSVLogger
• Logs: all logs (checkpoints, configs, etc.) are stored in a dynamically generated folder structure
• Hyperparameter Search: simple search is effortless with Hydra plugins like Optuna Sweeper
• Tests: generic, easy-to-adapt smoke tests for speeding up the development
• Continuous Integration: automatically test and lint your repo with Github Actions
• Best Practices: a couple of recommended tools, practices and standards

You have to do this before doing anything

# install requirements
pip install -r requirements.txt

Template contains example with MNIST classification.
When running python src/train.py you should see something like this:

python train.py trainer.max_epochs=20 model.optimizer.lr=1e-4

python train.py +model.new_param="owo"

# train with pytorch native automatic mixed precision (AMP)
python train.py trainer=gpu +trainer.precision=16

# set project and entity names in `configs/logger/wandb`
wandb:
  project: "your_project_name"
  entity: "your_wandb_team_name"

# train model with Weights&Biases (link to wandb dashboard should appear in the terminal)
python train.py logger=wandb

MIT License

Copyright (c) 2021 ashleve

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.