Adding Lora implementation for nn.Conv1d #2333

CCLDArjun · 2025-01-17T20:18:26Z

Resolves #2241

My comment shows that the shapes match in Enformer model: #2241 (comment)

Unsure how to test further it other than to run it in some training loop

BenjaminBossan

Thanks for adding a Conv1d implementation for LoRA. In general, this looks good, I have a few small comments, please check. Please also run make style to satisfy the linter.

Before merging, however, let's ensure that the code works correctly by adding some tests. We already have a "test factory" for the different LoRA layer types, so this is a matter of adding an entry for Conv1d. To do this, look at this code:

peft/tests/test_custom_models.py

Lines 877 to 931 in aa3f41f

    
           class ModelMha(nn.Module): 
        
               def __init__(self): 
        
                   super().__init__() 
        
                   self.mha = nn.MultiheadAttention(10, 2) 
        
                   self.lin0 = nn.Linear(10, 2) 
        
                   self.sm = nn.LogSoftmax(dim=-1) 
        
               def forward(self, X): 
        
                   X = X.float() 
        
                   X, _ = self.mha(X, X, X) 
        
                   X = self.lin0(X) 
        
                   X = self.sm(X) 
        
                   return X 
        
           class MockTransformerWrapper: 
        
               """Mock class to behave like a transformers model. 
        
               This is needed because the tests initialize the model by calling transformers_class.from_pretrained. 
        
               """ 
        
               @classmethod 
        
               def from_pretrained(cls, model_id, torch_dtype=None): 
        
                   # set the seed so that from_pretrained always returns the same model 
        
                   torch.manual_seed(0) 
        
                   if torch_dtype is None: 
        
                       torch_dtype = torch.float32 
        
                   if model_id == "MLP": 
        
                       return MLP().to(torch_dtype) 
        
                   if model_id == "EmbConv1D": 
        
                       return ModelEmbConv1D().to(torch_dtype) 
        
                   if model_id == "Conv2d": 
        
                       return ModelConv2D().to(torch_dtype) 
        
                   if model_id == "Conv3d": 
        
                       return ModelConv3D().to(torch_dtype) 
        
                   if model_id == "MLP_LayerNorm": 
        
                       return MLP_LayerNorm().to(torch_dtype) 
        
                   if model_id == "MLP2": 
        
                       return MLP2().to(torch_dtype) 
        
                   if model_id == "Conv2d2": 
        
                       return ModelConv2D2().to(torch_dtype) 
        
                   if model_id == "MHA": 
        
                       return ModelMha().to(torch_dtype) 
        
                   raise ValueError(f"model_id {model_id} not implemented")

What we need is to add a model similar to ModelMha but using Conv1d instead. The shape of the input should be 10. The from_pretrained method should get an update to dispatch to said model.

After this, it's only a matter of adding a row to the test cases, following this format:

peft/tests/test_custom_models.py

Lines 106 to 113 in aa3f41f

    
           ("Conv2d 1 LoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d"]}), 
        
           ("Conv2d 2 LoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d", "lin0"]}), 
        
           ("Conv2d 1 LoRA with DoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d"], "use_dora": True}), 
        
           ("Conv2d 2 LoRA with DoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d", "lin0"], "use_dora": True}), 
        
           ("Conv3d 1 LoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d"]}), 
        
           ("Conv3d 2 LoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d", "lin0"]}), 
        
           ("Conv3d 1 LoRA with DoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d"], "use_dora": True}), 
        
           ("Conv3d 2 LoRA with DoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d", "lin0"], "use_dora": True}),

I hope this makes sense. LMK if you have questions.

src/peft/tuners/lora/layer.py

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

CCLDArjun · 2025-01-22T23:00:20Z

make style was editing 22 different files so I ran ruff on my changed files directly I think it should be good

BenjaminBossan · 2025-01-23T10:37:22Z

make style was editing 22 different files so I ran ruff on my changed files directly I think it should be good

I think the most likely explanation is that you were using a different ruff version from what is used on CI. This would explain why CI still fails. Could you please ensure that the same version is used: ruff-0.6.9?

CCLDArjun · 2025-01-23T18:17:43Z

@BenjaminBossan Yep, 0.6.9 works much better make style is happy

BenjaminBossan

Thanks a lot for the updates, the PR LGTM. I tested it Hubert to have a more realistic test and it worked too (with the exception being the groups argument, but that is yet to be added by all conv layers).

Before merging, however, I just noticed one small change still needed, namely this error message which lists all supported layer types for LoRA:

peft/src/peft/tuners/lora/model.py

Lines 347 to 351 in bbb1128

    
           raise ValueError( 
        
               f"Target module {target} is not supported. Currently, only the following modules are supported: " 
        
               "`torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `torch.nn.Conv3d`, " 
        
               "`transformers.pytorch_utils.Conv1D`, `torch.nn.MultiheadAttention.`." 
        
           )

HuggingFaceDocBuilderDev · 2025-01-27T10:15:05Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

BenjaminBossan

Thanks for adding LoRA support for Conv1d, LGTM.

Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

use _ConvNd for conv1d lora

1f7b9fb

BenjaminBossan requested changes Jan 20, 2025

View reviewed changes

src/peft/tuners/lora/layer.py Outdated Show resolved Hide resolved

src/peft/tuners/lora/layer.py Show resolved Hide resolved

CCLDArjun and others added 2 commits January 22, 2025 14:06

Comment

bae612f

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

raise not implemented error for conv1d dora

ba03b7a

CCLDArjun force-pushed the conv1d branch from 7218296 to ba03b7a Compare January 22, 2025 22:46

add tests

84e14f4

CCLDArjun requested a review from BenjaminBossan January 22, 2025 23:00

style

944b4d1

BenjaminBossan requested changes Jan 24, 2025

View reviewed changes

update error messages

ee586ba

CCLDArjun requested a review from BenjaminBossan January 24, 2025 23:11

BenjaminBossan approved these changes Jan 27, 2025

View reviewed changes

BenjaminBossan merged commit f4176a9 into huggingface:main Jan 27, 2025
14 checks passed

Guy-Bilitski pushed a commit to Guy-Bilitski/peft that referenced this pull request May 13, 2025

ENH Add LoRA implementation for nn.Conv1d (huggingface#2333)

d4343cb

cyyever pushed a commit to cyyever/peft that referenced this pull request Sep 4, 2025

Bump liger-kernel to fix grad acc and more features (huggingface#2333)

c86b51c

Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adding Lora implementation for nn.Conv1d #2333

Adding Lora implementation for nn.Conv1d #2333

Uh oh!

CCLDArjun commented Jan 17, 2025

Uh oh!

BenjaminBossan left a comment

Uh oh!

Uh oh!

Uh oh!

CCLDArjun commented Jan 22, 2025

Uh oh!

BenjaminBossan commented Jan 23, 2025

Uh oh!

CCLDArjun commented Jan 23, 2025

Uh oh!

BenjaminBossan left a comment

Uh oh!

HuggingFaceDocBuilderDev commented Jan 27, 2025

Uh oh!

BenjaminBossan left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	class ModelMha(nn.Module):
	def __init__(self):
	super().__init__()
	self.mha = nn.MultiheadAttention(10, 2)
	self.lin0 = nn.Linear(10, 2)
	self.sm = nn.LogSoftmax(dim=-1)

	def forward(self, X):
	X = X.float()
	X, _ = self.mha(X, X, X)
	X = self.lin0(X)
	X = self.sm(X)
	return X


	class MockTransformerWrapper:
	"""Mock class to behave like a transformers model.

	This is needed because the tests initialize the model by calling transformers_class.from_pretrained.

	"""

	@classmethod
	def from_pretrained(cls, model_id, torch_dtype=None):
	# set the seed so that from_pretrained always returns the same model
	torch.manual_seed(0)

	if torch_dtype is None:
	torch_dtype = torch.float32

	if model_id == "MLP":
	return MLP().to(torch_dtype)

	if model_id == "EmbConv1D":
	return ModelEmbConv1D().to(torch_dtype)

	if model_id == "Conv2d":
	return ModelConv2D().to(torch_dtype)

	if model_id == "Conv3d":
	return ModelConv3D().to(torch_dtype)

	if model_id == "MLP_LayerNorm":
	return MLP_LayerNorm().to(torch_dtype)

	if model_id == "MLP2":
	return MLP2().to(torch_dtype)

	if model_id == "Conv2d2":
	return ModelConv2D2().to(torch_dtype)

	if model_id == "MHA":
	return ModelMha().to(torch_dtype)

	raise ValueError(f"model_id {model_id} not implemented")

	("Conv2d 1 LoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d"]}),
	("Conv2d 2 LoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d", "lin0"]}),
	("Conv2d 1 LoRA with DoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d"], "use_dora": True}),
	("Conv2d 2 LoRA with DoRA", "Conv2d", LoraConfig, {"target_modules": ["conv2d", "lin0"], "use_dora": True}),
	("Conv3d 1 LoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d"]}),
	("Conv3d 2 LoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d", "lin0"]}),
	("Conv3d 1 LoRA with DoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d"], "use_dora": True}),
	("Conv3d 2 LoRA with DoRA", "Conv3d", LoraConfig, {"target_modules": ["conv3d", "lin0"], "use_dora": True}),

	raise ValueError(
	f"Target module {target} is not supported. Currently, only the following modules are supported: "
	"`torch.nn.Linear`, `torch.nn.Embedding`, `torch.nn.Conv2d`, `torch.nn.Conv3d`, "
	"`transformers.pytorch_utils.Conv1D`, `torch.nn.MultiheadAttention.`."
	)

Adding Lora implementation for nn.Conv1d #2333

Adding Lora implementation for nn.Conv1d #2333

Uh oh!

Conversation

CCLDArjun commented Jan 17, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

CCLDArjun commented Jan 22, 2025

Uh oh!

BenjaminBossan commented Jan 23, 2025

Uh oh!

CCLDArjun commented Jan 23, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jan 27, 2025

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants