added support for Conv1d for DoRA #2531
Conversation
BenjaminBossan
left a comment
There was a problem hiding this comment.
Thanks for working on the support of nn.Conv1d* for DoRA. Before proceeding, let's add unit tests for this feature. This is as simple as adding a line like this:
("Conv1d LoRA Conv1d with DoRA", "Conv1d", LoraConfig, {"target_modules": ["conv1d"], "use_dora": True}),
here:
peft/tests/test_custom_models.py
Line 115 in 6c054d0
*Just to be clear, we already support transformers.pytorch_utils.Conv1D, which is a different layer type.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
Thanks, could you please run |
|
@EskildAndersen please make sure to use the same ruff version as specified in either run or |
This reverts commit eaed5bc.
githubnemo
left a comment
There was a problem hiding this comment.
LGTM. I suggest to not delete the clarifying comment about the handling of Conv layers but otherwise I think that this can be merged. I'll add the suggestion and once the pipeline is green, I'll merge it.
Thanks for the contribution, good work :)
I took over this PR and the comments were adressed.
DoRA now supports Conv1d layers and, notably, the check for how to deal with other than linear layers was softened from checking for 4 dimensions to now 3 dimensions since `Conv1d` layers have 3 elements instead of 4.
…2531) Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
No description provided.