Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Positive RDMA / smb-direct „taskman test“ between Win-client and linux-Server? #488

Open
besterino opened this issue Nov 2, 2024 · 3 comments

Comments

@besterino
Copy link

Hi!

Since I am struggling and cannot get it to work: has anyone successfully tested RDMA/smb-direct between a linux ksmbd server and a windows client, so that the windows taskmanager on the client does not show corresponding load (load between windows machine is zero or at least very low (kb/s)) while copying large files (e.g. 30GB) at much higher speeds of 1-2.5GB/s?

If so, could you please give some guidance on hardware, distro, specific steps, settings etc. used?

I have described my tries also here in another thread, but willing to start from scratch if there’s a setup / route more likely / proven to succeed:

#466 (comment)

@namjaejeon
Copy link
Owner

Have you ever searched RDMA or smb-direct work in ISSUE of cifsd-team github?

cifsd-team#542
cifsd-team#604
and more... in ISSUE of cifsd-team github.

If so, could you please give some guidance on hardware, distro, specific steps, settings etc. used?

You should add "server multi channel support = yes" parameter in [global] section of your ksmbd.conf
and build ksmbd.ko after turning CONFIG_SMB_SERVER_SMBDIRECT config on.
The smb-direct(RDMA) feature is that the server responds from the client request. So please refer to your client settings or guide.

If it still doesn't work, please let me know.

@besterino
Copy link
Author

Thank you for the input.

Yes, I had a look at various posts but still could not get it to work. Another one I found interesting but also without success: https://forum.level1techs.com/t/how-can-i-help-with-the-new-truenas-100g-testing/179052/8

As to my ksmbd.conf, it already has/had "server multi channel support = yes".
My kernel was build with CONFIG_SMB_SERVER_SMBDIRECT enabled, at least according to /boot/config-6.11.0-9-generic.

On the windows client I tracked RDMA activity with Perfmon. It apparently tries to establish RDMA connections, but they fail.

The only time I see any smb_direct messages in dmesg is immediately after start of the service:
[ +7.774154] ksmbd: selected SMB3_11 dialect idx = 3
[ +0.000009] ksmbd: selected SMB3_11 dialect idx = 3
[ +0.000179] ksmbd: smb_direct: ib device added: name rocep33s0f0
[ +0.000002] ksmbd: smb_direct: ib device added: name rocep33s0f1
[ +0.000354] ksmbd: smb_direct: init RDMA listener. cm_id=0000000084cd3fdd

@namjaejeon
Copy link
Owner

The only time I see any smb_direct messages in dmesg is immediately after start of the service:

Is there any error messages from ksmbd: smb_direct: ? This message("ksmbd: smb_direct: init RDMA listener. cm_id=0000000084cd3fdd") is the last one ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants