10gbe bandwidth issues

Hi guys, battling some 10gbe issues that I am having in Manjaro.
I am running a Mellanox Connect-X3 in my system and that is running to a Microtik Switch that is then connected to a Dell R730XD.
In Windows I can transfer files back and forth between the desktop and the server at 950MB/s without issues both directions

In Manjaro the speed is capped at 120MB/s no matter the direction.
Not sure how to go about diagnosing and what steps to take.

My first thought was too try the Linux driver for the Mellanox Connect-X3.
These are the Linux Drivers they list.
connect X3

Not sure where to head from there.

Please post your system info as outlined here:

Also please post the output of:

lspci | grep Network
  Kernel: 5.10.41-1-MANJARO x86_64 bits: 64 compiler: gcc v: 11.1.0 
  parameters: BOOT_IMAGE=/boot/vmlinuz-5.10-x86_64 
  root=UUID=b87bd145-97e0-4bd3-aecd-1cd1675587cb rw quiet splash apparmor=1 
  security=apparmor resume=UUID=ed39a145-b614-4873-a477-85de40d529d8 
  Desktop: GNOME 3.38.4 tk: GTK 3.24.29 wm: gnome-shell dm: GDM 40.0 
  Distro: Manjaro Linux base: Arch Linux 
  Type: Desktop System: EVGA product: X299 DARK v: N/A serial: <filter> 
  Chassis: type: 3 serial: <filter> 
  Mobo: EVGA model: X299 DARK v: 1.0 serial: <filter> 
  UEFI: American Megatrends v: 1.26 date: 01/20/2021 
  ID-1: hidpp_battery_0 charge: N/A condition: N/A volts: 4.0 min: N/A 
  model: Logitech G Pro Wireless Gaming Mouse type: N/A serial: <filter> 
  status: Discharging 
  RAM: total: 31.06 GiB used: 2.06 GiB (6.6%) 
  RAM Report: permissions: Unable to run dmidecode. Root privileges required. 
  Info: 14-Core model: Intel Core i9-10940X bits: 64 type: MT MCP 
  arch: Cascade Lake family: 6 model-id: 55 (85) stepping: 7 
  microcode: 5003006 cache: L2: 19.2 MiB bogomips: 184883 
  Speed: 1200 MHz min/max: 1200/4700 MHz Core speeds (MHz): 1: 1200 2: 1202 
  3: 1200 4: 1200 5: 1200 6: 1200 7: 1201 8: 1200 9: 1200 10: 1201 11: 1200 
  12: 1201 13: 1200 14: 1200 15: 1201 16: 3299 17: 1200 18: 1200 19: 1200 
  20: 1200 21: 1200 22: 1200 23: 1200 24: 1200 25: 1202 26: 1199 27: 2196 
  28: 1200 
  Flags: 3dnowprefetch abm acpi adx aes aperfmperf apic arat arch_capabilities 
  arch_perfmon art avx avx2 avx512_vnni avx512bw avx512cd avx512dq avx512f 
  avx512vl bmi1 bmi2 bts cat_l3 cdp_l3 clflush clflushopt clwb cmov 
  constant_tsc cpuid cpuid_fault cqm cqm_llc cqm_mbm_local cqm_mbm_total 
  cqm_occup_llc cx16 cx8 dca de ds_cpl dtes64 dtherm dts epb erms est f16c 
  flush_l1d fma fpu fsgsbase fxsr ht ibpb ibrs ibrs_enhanced ida intel_pt 
  invpcid invpcid_single lahf_lm lm mba mca mce md_clear mmx monitor movbe mpx 
  msr mtrr nonstop_tsc nopl nx pae pat pbe pcid pclmulqdq pdcm pdpe1gb pebs 
  pge pln pni popcnt pse pse36 pts rdrand rdseed rdt_a rdtscp rep_good sdbg 
  sep smap smep ss ssbd sse sse2 sse4_1 sse4_2 ssse3 stibp syscall tm tm2 tsc 
  tsc_adjust tsc_deadline_timer vme x2apic xgetbv1 xsave xsavec xsaveopt 
  xsaves xtopology xtpr 
  Vulnerabilities: Type: itlb_multihit status: KVM: VMX unsupported 
  Type: l1tf status: Not affected 
  Type: mds status: Not affected 
  Type: meltdown status: Not affected 
  Type: spec_store_bypass 
  mitigation: Speculative Store Bypass disabled via prctl and seccomp 
  Type: spectre_v1 
  mitigation: usercopy/swapgs barriers and __user pointer sanitization 
  Type: spectre_v2 mitigation: Enhanced IBRS, IBPB: conditional, RSB filling 
  Type: srbds status: Not affected 
  Type: tsx_async_abort mitigation: TSX disabled 
  Device-1: NVIDIA GA102 [GeForce RTX 3090] vendor: eVga.com. driver: nvidia 
  v: 465.31 alternate: nouveau,nvidia_drm bus-ID: 65:00.0 chip-ID: 10de:2204 
  class-ID: 0300 
  Display: x11 server: X.org 1.20.11 compositor: gnome-shell driver: 
  loaded: nvidia resolution: <missing: xdpyinfo> 
  OpenGL: renderer: NVIDIA GeForce RTX 3090/PCIe/SSE2 v: 4.6.0 NVIDIA 465.31 
  direct render: Yes 
  Device-1: Intel 200 Series PCH HD Audio vendor: eVga.com. 
  driver: snd_hda_intel v: kernel bus-ID: 00:1f.3 chip-ID: 8086:a2f0 
  class-ID: 0403 
  Device-2: NVIDIA vendor: eVga.com. driver: snd_hda_intel v: kernel 
  bus-ID: 65:00.1 chip-ID: 10de:1aef class-ID: 0403 
  Sound Server-1: ALSA v: k5.10.41-1-MANJARO running: yes 
  Sound Server-2: JACK v: 0.125.0 running: no 
  Sound Server-3: PulseAudio v: 14.2 running: yes 
  Sound Server-4: PipeWire v: 0.3.28 running: no 
  Device-1: Mellanox MT27500 Family [ConnectX-3] driver: mlx4_core v: 4.0-0 
  port: 0580 bus-ID: 02:00.0 chip-ID: 15b3:1003 class-ID: 0200 
  IF: enp2s0 state: up speed: 10000 Mbps duplex: full mac: <filter> 
  IP v4: <filter> type: noprefixroute scope: global broadcast: <filter> 
  IP v6: <filter> type: noprefixroute scope: link 
  WAN IP: <filter> 
  Message: No bluetooth data found. 
  Message: No logical block device data found. 
  Message: No RAID data found. 
  Local Storage: total: 4.77 TiB used: 32.11 GiB (0.7%) 
  SMART Message: Required tool smartctl not installed. Check --recommends 
  ID-1: /dev/nvme0n1 maj-min: 259:0 vendor: Intel model: SSDPEKNW020T8 
  size: 1.86 TiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s 
  lanes: 4 rotation: SSD serial: <filter> rev: 002C temp: 34.9 C scheme: GPT 
  ID-2: /dev/nvme1n1 maj-min: 259:5 vendor: Intel model: SSDPEKNW020T8 
  size: 1.86 TiB block-size: physical: 512 B logical: 512 B speed: 31.6 Gb/s 
  lanes: 4 rotation: SSD serial: <filter> rev: 002C temp: 32.9 C scheme: GPT 
  ID-3: /dev/sda maj-min: 8:0 vendor: SanDisk model: SDSSDH31024G 
  size: 953.87 GiB block-size: physical: 512 B logical: 512 B speed: 6.0 Gb/s 
  rotation: SSD serial: <filter> rev: 4000 scheme: GPT 
  ID-4: /dev/sdb maj-min: 8:16 type: USB vendor: Samsung model: Flash Drive 
  size: 119.51 GiB block-size: physical: 512 B logical: 512 B rotation: SSD 
  serial: <filter> rev: 1100 scheme: GPT 
  Message: No optical or floppy data found. 
  ID-1: / raw-size: 919.41 GiB size: 903.9 GiB (98.31%) used: 10.43 GiB (1.2%) 
  fs: ext4 dev: /dev/sda2 maj-min: 8:2 label: N/A 
  uuid: b87bd145-97e0-4bd3-aecd-1cd1675587cb 
  ID-2: /boot/efi raw-size: 300 MiB size: 299.4 MiB (99.80%) 
  used: 296 KiB (0.1%) fs: vfat dev: /dev/sda1 maj-min: 8:1 label: NO_LABEL 
  uuid: 564A-2D62 
  ID-3: /run/media/sean/VTOYEFI raw-size: 32 MiB size: 31.9 MiB (99.75%) 
  used: 25.2 MiB (78.8%) fs: vfat dev: /dev/sdb2 maj-min: 8:18 label: VTOYEFI 
  uuid: 133B-CFB9 
  ID-4: /run/media/sean/Ventoy raw-size: 119.48 GiB size: 119.48 GiB (100.00%) 
  used: 21.65 GiB (18.1%) fs: exfat dev: /dev/sdb1 maj-min: 8:17 label: Ventoy 
  uuid: 4E21-0000 
  Kernel: swappiness: 60 (default) cache-pressure: 100 (default) 
  ID-1: swap-1 type: partition size: 34.16 GiB used: 0 KiB (0.0%) priority: -2 
  dev: /dev/sda3 maj-min: 8:3 label: N/A 
  uuid: ed39a145-b614-4873-a477-85de40d529d8 
  ID-1: /dev/nvme0n1p1 maj-min: 259:1 size: 100 MiB fs: vfat label: N/A 
  uuid: BAC8-F110 
  ID-2: /dev/nvme0n1p2 maj-min: 259:2 size: 16 MiB fs: <superuser required> 
  label: N/A uuid: N/A 
  ID-3: /dev/nvme0n1p3 maj-min: 259:3 size: 1.86 TiB fs: ntfs label: N/A 
  uuid: 04A2CF4AA2CF3EC4 
  ID-4: /dev/nvme0n1p4 maj-min: 259:4 size: 499 MiB fs: ntfs label: N/A 
  uuid: 58E0D342E0D32552 
  ID-5: /dev/nvme1n1p1 maj-min: 259:6 size: 16 MiB fs: <superuser required> 
  label: N/A uuid: N/A 
  ID-6: /dev/nvme1n1p2 maj-min: 259:7 size: 1.86 TiB fs: ntfs label: Games 
  uuid: 88EE3246EE322CB6 
  Hub-1: 1-0:1 info: Full speed (or root) Hub ports: 16 rev: 2.0 
  speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900 
  Device-1: 1-2:2 info: MCT Elektronikladen MPS type: HID 
  driver: hid-generic,usbhid interfaces: 1 rev: 1.1 speed: 1.5 Mb/s 
  power: 10mA chip-ID: 0c70:f003 class-ID: 0300 
  Device-2: 1-5:3 info: MCT Elektronikladen aquaero type: Keyboard,HID 
  driver: hid-generic,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s 
  power: 100mA chip-ID: 0c70:f001 class-ID: 0300 serial: <filter> 
  Device-3: 1-7:4 info: Corsair K95 RGB Platinum Keyboard [RGP0056] 
  type: Keyboard,HID driver: hid-generic,usbhid interfaces: 2 rev: 2.0 
  speed: 12 Mb/s power: 500mA chip-ID: 1b1c:1b2d class-ID: 0300 
  serial: <filter> 
  Device-4: 1-9:5 info: Logitech USB Receiver type: Keyboard,Mouse,HID 
  driver: logitech-djreceiver,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s 
  power: 98mA chip-ID: 046d:c539 class-ID: 0300 
  Device-5: 1-12:6 info: MCT Elektronikladen aquaero type: Keyboard,HID 
  driver: hid-generic,usbhid interfaces: 3 rev: 2.0 speed: 12 Mb/s 
  power: 100mA chip-ID: 0c70:f001 class-ID: 0300 serial: <filter> 
  Hub-2: 2-0:1 info: Full speed (or root) Hub ports: 10 rev: 3.0 speed: 5 Gb/s 
  chip-ID: 1d6b:0003 class-ID: 0900 
  Hub-3: 3-0:1 info: Full speed (or root) Hub ports: 2 rev: 2.0 
  speed: 480 Mb/s chip-ID: 1d6b:0002 class-ID: 0900 
  Hub-4: 4-0:1 info: Full speed (or root) Hub ports: 2 rev: 3.1 speed: 10 Gb/s 
  chip-ID: 1d6b:0003 class-ID: 0900 
  Device-1: 4-2:2 info: Silicon Motion - Taiwan (formerly Feiya ) Flash Drive 
  type: Mass Storage driver: usb-storage interfaces: 1 rev: 3.1 speed: 5 Gb/s 
  power: 304mA chip-ID: 090c:1000 class-ID: 0806 serial: <filter> 
  System Temperatures: cpu: 35.0 C mobo: N/A gpu: nvidia temp: 27 C 
  Fan Speeds (RPM): N/A gpu: nvidia fan: 0% 
  Processes: 466 Uptime: N/A wakeups: 2 Init: systemd v: 247 tool: systemctl 
  Compilers: gcc: 11.1.0 Packages: pacman: 1166 lib: 312 flatpak: 0 Shell: Zsh 
  v: 5.8 running-in: gnome-terminal inxi: 3.3.04

I get no output when entering lspci | grep Network

You already have the correct driver installed provided by the kernel.

It is a PCI NIC, right? Try:

lspci | grep Mellanox

Also see: InfiniBand - ArchWiki

I updated the firmware as stated in the arch Wiki.
I am not using the controller in infiniband mode.

lspci | grep Mellanox                                            ✔ 
02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Image type:            FS2
FW Version:            2.42.5000
FW Release Date:       5.9.2017
Product Version:
Rom Info:              type=PXE version=3.4.752
Device ID:             4099
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff 
MACs:                                       e41d2d20c700     e41d2d20c701
PSID:                  MT_1170110023

I feel in arch the card is defaulting to infiniband and in Windows I have it setup for Ethernet.
Unfortunately this means installing the driver in Arch which means probably building it from source which currently is above my technical skills.

You can check following these instructions:

In order to change the port mode without the script

Again, it’s already installed.

Getting out of my league on configuring stuff.
Trying to run that inbox driver one I just get permission denied even when using sudo and the correct device ID path.
The driver is installed but not the MLNX_OFED driver.

Please post terminal output, not screenshots.

You may need to explore the folder there as the device port may be different.

OK sorry about that.
In that folder but man that is going to take a while to go through all of them to find out which on is the right one.

Actually think I found it and it shows it’s already in eth mode.

You should be checking what the ports are set to with cat before blindly attempting to modify them. Check those instructions again. You can’t modify what doesn’t exist. :wink:

Also you need root permissions to set that value.
Either switch to the root account before issuing the command (with su), or use tee with sudo:

echo eth | sudo tee /sys/...