Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Covert Conv3d to Conv2d, But output different #139066

Open
fengpings opened this issue Oct 28, 2024 · 4 comments
Open

Covert Conv3d to Conv2d, But output different #139066

fengpings opened this issue Oct 28, 2024 · 4 comments
Labels
module: convolution Problems related to convolutions (THNN, THCUNN, CuDNN) triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@fengpings
Copy link

🐛 Describe the bug

I tried to convert a Conv3d to Conv2d by splitting 3D kernels by the temporal axis, but outputs from the two differ:
image
image
I expect output1 and output2 should be the same. Any details I've overlooked?

code:

t = torch.randn(1, 1, 4, 4, dtype=torch.float32)
conv3d = torch.nn.Conv3d(out_channels=1, in_channels=1, kernel_size=(2,2,2), stride=(2,2,2), bias=False, dtype=torch.float32)
# conv3d2 = torch.nn.Conv3d(out_channels=1, in_channels=1, kernel_size=(2,2,2), stride=(2,2,2), bias=False, dtype=torch.bfloat16)
# conv3d.weight.data = torch.ones(1, 1, 2, 2, 2, dtype=torch.bfloat16)
# conv3d2.weight.data = conv3d.weight.data
conv2d1 = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2,2), stride=(2,2), bias=False, dtype=torch.float32)
conv2d2 = torch.nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(2,2), stride=(2,2), bias=False, dtype=torch.float32)
# conv2d3 = torch.nn.Conv2d(in_channels=3, out_channels=12, kernel_size=(14,14), stride=(14,14), bias=False)
# print(conv3d.weight.shape)
# conv2d1.weight = torch.nn.parameter.Parameter(conv3d.weight.detach().reshape(12, 6, 14, 14).contiguous())
# conv2d3.weight = torch.nn.parameter.Parameter(torch.mean(conv3d.weight.detach(), dim=2).squeeze(dim=2))

print(conv3d.weight.data.dtype)
conv2d1.weight.data = conv3d.weight.data[:,:, 0, :, :]
conv2d2.weight.data = conv3d.weight.data[:,:, 1, :, :]


input = torch.stack([t, t], dim=0).view(-1, 1, 2, 2, 2)
# print(input.shape)
output1 = conv3d(input).view(-1, 1)
output2 = (conv2d1(input[:, :, 0, :, :]) + conv2d2(input[:, :, 1, :, :])).view(-1, 1)
# output3 = conv3d2(input).view(-1, 1)
print(output1==output2)

Versions

PyTorch version: 2.4.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.0.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.26.3)
CMake version: version 3.28.1
Libc version: N/A

Python version: 3.10.13 (main, Sep 11 2023, 08:16:02) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.0.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Max

Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] torch==2.4.1
[pip3] torchaudio==2.4.1
[pip3] torchvision==0.19.1
[conda] numpy 1.26.3 pypi_0 pypi
[conda] torch 2.4.1 pypi_0 pypi
[conda] torchaudio 2.4.1 pypi_0 pypi
[conda] torchvision 0.19.1 pypi_0 pypi
@netra212
Copy link

Try this code:

`

import torch
import torch.nn as nn

torch.manual_seed(42)

t = torch.randn(1, 1, 4, 4, dtype=torch.float32)

print("\n3D-Tensors:\n\n", t)

input = torch.stack([t, t], dim=0).view(-1, 1, 2, 4, 4)  

class Convert2DTo2D(nn.Module):

def __init__(self, in_channels, out_channels, kernel_size, stride):

    super().__init__()

    # Initializing the Conv3D layer. 
    self.conv3d = nn.Conv3d(in_channels=in_channels,
                            out_channels=out_channels, 
                            kernel_size=(2,2,2), 
                            stride=(1,1,1), 
                            bias=False)

    # Initializing the Conv2d Layer.
    self.conv2d1 = nn.Conv2d(in_channels=in_channels, 
                             out_channels=out_channels, 
                             kernel_size=(2,2), 
                             stride=(1,1), 
                             bias=False)

    self.conv2d2 = nn.Conv2d(in_channels=in_channels, 
                        out_channels=out_channels, 
                        kernel_size=(2,2), 
                        stride=(1,1), 
                        bias=False)
  
    # This is the context-manager which disables the gradient calculation while inferencing reducing the consumption of the memory for faster computations. 
    with torch.no_grad():
        self.conv2d1.weight.data = self.conv3d.weight.data[:,:,0,:,:]
        self.conv2d2.weight.data = self.conv3d.weight.data[:,:,1,:,:]

def forward(self, x):
    output1 = self.conv3d(x)

    # split input along temporal axis and apply Conv2d. 
    output2 = (self.conv2d1(x[:,:,0,:,:]) + self.conv2d2(x[:,:,1,:,:])).unsqueeze(2)

    # Combine the outputs into the same shape as Conv3D.
    return output1, output2

model = Convert2DTo2D(in_channels=1, out_channels=1, kernel_size=(2, 2, 2), stride=(1, 1, 1))

output1, output2 = model(input)

print(output1)

print(output2)

print(f"\nInput shape: {input.shape}\n")
print(f"\nShape of Output1: {output1.shape}\n")
print(f"\nShape of Output2: {output2.shape}\n")`

Screenshot 2024-10-28 at 16 52 10

@fengpings
Copy link
Author

Try this code:

`

import torch
import torch.nn as nn

torch.manual_seed(42)

t = torch.randn(1, 1, 4, 4, dtype=torch.float32)

print("\n3D-Tensors:\n\n", t)

input = torch.stack([t, t], dim=0).view(-1, 1, 2, 4, 4)  

class Convert2DTo2D(nn.Module):

def __init__(self, in_channels, out_channels, kernel_size, stride):

    super().__init__()

    # Initializing the Conv3D layer. 
    self.conv3d = nn.Conv3d(in_channels=in_channels,
                            out_channels=out_channels, 
                            kernel_size=(2,2,2), 
                            stride=(1,1,1), 
                            bias=False)

    # Initializing the Conv2d Layer.
    self.conv2d1 = nn.Conv2d(in_channels=in_channels, 
                             out_channels=out_channels, 
                             kernel_size=(2,2), 
                             stride=(1,1), 
                             bias=False)

    self.conv2d2 = nn.Conv2d(in_channels=in_channels, 
                        out_channels=out_channels, 
                        kernel_size=(2,2), 
                        stride=(1,1), 
                        bias=False)
  
    # This is the context-manager which disables the gradient calculation while inferencing reducing the consumption of the memory for faster computations. 
    with torch.no_grad():
        self.conv2d1.weight.data = self.conv3d.weight.data[:,:,0,:,:]
        self.conv2d2.weight.data = self.conv3d.weight.data[:,:,1,:,:]

def forward(self, x):
    output1 = self.conv3d(x)

    # split input along temporal axis and apply Conv2d. 
    output2 = (self.conv2d1(x[:,:,0,:,:]) + self.conv2d2(x[:,:,1,:,:])).unsqueeze(2)

    # Combine the outputs into the same shape as Conv3D.
    return output1, output2

model = Convert2DTo2D(in_channels=1, out_channels=1, kernel_size=(2, 2, 2), stride=(1, 1, 1))

output1, output2 = model(input)

print(output1)

print(output2)

print(f"\nInput shape: {input.shape}\n")
print(f"\nShape of Output1: {output1.shape}\n")
print(f"\nShape of Output2: {output2.shape}\n")`

Screenshot 2024-10-28 at 16 52 10

image thanks but the problem still exists. Is this due to the implementation of Conv3D?

@ezyang
Copy link
Contributor

ezyang commented Oct 29, 2024

It's unlikely to be a conv3d problem. I'd suggest asking about this at https://discuss.pytorch.org/ and coming back here if you have more evidence that this is specifically a PyTorch bug

@ezyang ezyang added module: convolution Problems related to convolutions (THNN, THCUNN, CuDNN) triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Oct 29, 2024
@fengpings
Copy link
Author

It's unlikely to be a conv3d problem. I'd suggest asking about this at https://discuss.pytorch.org/ and coming back here if you have more evidence that this is specifically a PyTorch bug

thanks, I'll post it there. Any thoughts on what the cause might be?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
module: convolution Problems related to convolutions (THNN, THCUNN, CuDNN) triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ezyang @fengpings @netra212