[docs] Task guide for semantic segmentation with LoRA #307
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The documentation is not available anymore as the PR was closed or merged. |
pacman100
approved these changes
Apr 14, 2023
Contributor
pacman100
left a comment
There was a problem hiding this comment.
Thank you @MKhalusova for this nice usage guide on semantic segmentation using lora! ✨
Left a comment.
cc @sayakpaul for a quick look as he had worked on this example
Comment on lines
+254
to
+277
| Let's review the `LoraConfig`. To enable LoRA technique, we must define the target modules within `LoraConfig` so that | ||
| `PeftModel` can update the necessary matrices. Specifically, we want to target the `query` and `value` matrices in the | ||
| attention blocks of the base model. These matrices are identified by their respective names, "query" and "value". | ||
| Therefore, we should specify these names in the `target_modules` argument of `LoraConfig`. | ||
|
|
||
| After we wrap our base model `model` with `PeftModel` along with the config, we get | ||
| a new model where only the LoRA parameters are trainable (so-called "update matrices") while the pre-trained parameters | ||
| are kept frozen. These include the parameters of the randomly initialized classifier parameters too. This is NOT we want | ||
| when fine-tuning the base model on our custom dataset. To ensure that the classifier parameters are also trained, we | ||
| specify `modules_to_save`. This also ensures that these modules are serialized alongside the LoRA trainable parameters | ||
| when using utilities like `save_pretrained()` and `push_to_hub()`. | ||
|
|
||
| In addition to specifying the `target_modules` within `LoraConfig`, we also need to specify the `modules_to_save`. When | ||
| we wrap our base model with `PeftModel` and pass the configuration, we obtain a new model in which only the LoRA parameters | ||
| are trainable, while the pre-trained parameters and the randomly initialized classifier parameters are kept frozen. | ||
| However, we do want to train the classifier parameters. By specifying the `modules_to_save` argument, we ensure that the | ||
| classifier parameters are also trainable, and they will be serialized alongside the LoRA trainable parameters when we | ||
| use utility functions like `save_pretrained()` and `push_to_hub()`. | ||
|
|
||
| Let's review the rest of the parameters: | ||
|
|
||
| - `r`: The dimension used by the LoRA update matrices. | ||
| - `alpha`: Scaling factor. | ||
| - `bias`: Specifies if the `bias` parameters should be trained. `None` denotes none of the `bias` parameters will be trained. |
Contributor
There was a problem hiding this comment.
insightful and easy to follow!
sayakpaul
approved these changes
Apr 14, 2023
Member
sayakpaul
left a comment
There was a problem hiding this comment.
Oh we are going hf.co/docs/peft. Wow, that is so cool!
Let's ship this! 🚀
Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. |
Guy-Bilitski
pushed a commit
to Guy-Bilitski/peft
that referenced
this pull request
May 13, 2025
* WIP: semantic segmentation example * make style * some polishing * Update docs/source/task_guides/semantic_segmentation_lora.mdx Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com> * minor update --------- Co-authored-by: Sourab Mangrulkar <13534540+pacman100@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Moves LoRA for semantic segmentation notebook into the docs.