Mellanox Spectrum SN2000 Series

The Mellanox Spectrum SN2000 Series Switches are Managed Ethernet Switches with a maximum speed of 100Gb/s and up to 56 ports.
They use the switchdev API to offload the kernel's forwarding plane to its ASICs, allowing for more network throughput.
More info: https://github.com/Mellanox/mlxsw/wiki

Generic Router

Unfortunately, there is currently no pre-built OpenWrt image for this target in the Firmware Selector. There are two options to obtain an image and set up a working OpenWrt system on this device:

  • You can build your image manually following the instructions described here.
  • You can also obtain a generic x86_64 snapshot image from the Firmware Selector here. After you have installed the image on the device and after every sysupgrade, you need to install the required packages as described here.


Flashing the image is done as you do it with any other x86 target: https://openwrt.org/docs/guide-user/installation/openwrt_x86#installation

This will erase the complete disk and flash OpenWrt, removing existing operating systems, including the commonly pre-installed ONIE or Onyx.


The instructions on this page will show you how to install OpenWrt on the switch by flashing it from a USB device. You have to connect the host PC to the serial port of the network switch to navigate through the BIOS and the installation steps. The parameters of the serial connection can be found here.

The USB device will serve as a kind of Live USB. If you already have a Live USB from another Linux distribution, you can also use that and skip this step.


First, plug the USB device into your PC, identify your USB device (e.g., by running fdisk -l; the instructions here will use /dev/sdx), and flash the image to it. :!: This will erase all your data on the USB device.

gunzip openwrt-x86-64-generic-ext4-combined.img.gz

# Replace sdx with your device
sudo dd if=openwrt-x86-64-generic-ext4-combined.img of=/dev/sdx


Since the OpenWrt root partition size is only 104MB by default, we must enlarge it and its filesystem. :!: Some Linux distributions automatically mount the root partition, so you may have to unmount it before.

sudo parted -s /dev/sdx resizepart 2 100%
sudo resize2fs -f /dev/sdx2


After that, you can mount the root partition and copy the OpenWrt to the home directory of the user root.

sudo mkdir /mnt/usb
sudo mount /dev/sdx2 /mnt/usb/
cp openwrt-x86-64-generic-ext4-combined.img /mnt/usb/root/
sudo umount /dev/sdx2
sudo rmdir /mnt/usb


Next, (safely) unplug the USB device, plug it into the Mellanox switch, and connect your PC to its serial port. To enter the BIOS, reboot the switch and press CTRL+B while you see the BIOS information text (American Megatrends …).
The default password to enter the BIOS is admin.
Switch to the Save & Exit tab and select your USB device from the Boot Override options.
When booted, press Enter on your keyboard to activate the console and start flashing the image.

cd ~
dd if=openwrt-x86-64-mellanox_spectrum-sn2100-ext4-combined.img of=/dev/sda


Finally, if the flashing process is successful, reboot, and the switch should boot into the newly installed OpenWrt system.

As mentioned before, the root partition size is only 104 MB. To increase it and the filesystem, please follow the steps of the following link: https://openwrt.org/docs/guide-user/installation/openwrt_x86#expanding_root_partition_and_filesystem.

If you haven't used a manually built image for the installation, you must proceed with the following section to install all required packages.

The following steps are needed if you haven't built your image manually and used the generic x86_64 snapshot image from the Firmware Selector. You must also repeat those steps if you do a sysupgrade on the device.

First, plug in an ethernet cable into the management port and make sure that it has internet access. By default, the port has the IPv4 address 192.168.1.1/24, but you can change it by following the steps provided here: https://openwrt.org/docs/guide-user/network/network_configuration.

After you have gained internet access, you can install the required package through the opkg package manager. Before you can do that, update the list of available packages:

opkg update


After that, you can install the packages required for the QSFP ports and the LEDs to work:

opkg install mlxsw_spectrum-firmware kmod-mlxfw kmod-mlxsw-core kmod-mlxsw-i2c kmod-mlxsw-minimal kmod-mlxsw-pci kmod-mlxsw-spectrum kmod-leds-mlxcpld


If you also want to monitor the temperature of the CPU, RAM, and SSD drive, install the following additional packages:

opkg install kmod-hwmon-coretemp kmod-hwmon-drivetemp kmod-hwmon-jc42 kmod-i2c-i801


Finally, reboot the device, and the QSFP ports of the switch should be functional now.

If you want to build your image manually for this device, follow the steps provided on this page: https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem.

When you are inside the build system configuration interface (make menuconfig), select the following target configuration:

  • Target System: x86
  • Subtarget: x86_64
  • Target Profile: Generic x86/64

It isn't mandatory, but you should also unselect Use Console Terminal (in addition to Serial) under the Target Images section to avoid possible problems while using the serial console.


For the QSFP ports and the LEDs to work, you have to select the following packages:

  • Firmware: mlxsw_spectrum-firmware
  • Kernel modules → Network Devices: kmod-mlxfw, kmod-mlxsw-core, kmod-mlxsw-i2c, kmod-mlxsw-minimal, kmod-mlxsw-pci, kmod-mlxsw-spectrum
  • Kernel modules → LED modules: kmod-leds-mlxcpld


If you also want to monitor the temperature of the CPU, RAM, and SSD drive, you can select the following kernel modules:

  • Kernel modules → Hardware Monitoring Support: kmod-hwmon-coretemp, kmod-hwmon-drivetemp, kmod-hwmon-jc42
  • Kernel modules → I2C support: kmod-i2c-i801


After the build process (make) has finished, you can find the compiled image at bin/targets/x86/64/openwrt-x86-64-generic-ext4-combined.img.gz.

The instructions here use 192.168.1.1 as the IPv4 address of the switch. Replace it with your configured IP address where applicable.

It is important that you put the firmware image into the ramdisk (/tmp) before you start flashing.

sysupgrade

If you have built your image manually, transfer the image from your host machine to the switch first:

scp -O openwrt-x86-64-generic-ext4-combined.img.gz root@192.168.1.1:/tmp


Then, log in via SSH and enter the following command:

sysupgrade /tmp/openwrt-x86-64-generic-ext4-combined.img.gz



If you want to use the generic x86_64 snapshot image from the Firmware Selector, log in via SSH and enter the following commands.
:!: But remember that the QSFP ports won't work anymore after the switch has rebooted, so you must repeat the steps described here after the sysupgrade has finished.

cd /tmp
wget https://downloads.openwrt.org/snapshots/targets/x86/64/openwrt-x86-64-generic-ext4-combined.img.gz
sysupgrade /tmp/openwrt-x86-64-generic-ext4-combined.img.gz

The default network configuration is:

Interface Name Description Default configuration
br-lan Management Port 192.168.1.1/24
lo Loopback 127.0.0.1

The Mellanox Spectrum SN2000 Series has the following buttons:

BUTTON Event
Reset reset

Front:
Insert photo of front of the casing

Back:
Insert photo of back of the casing

Backside label:
Insert photo of backside label

Note: This will void your warranty!

FIXME Describe what needs to be done to open the device, e.g. remove rubber feet, adhesive labels, screws, ...

  • To remove the cover and open the device, do a/b/c

Main PCB:
Insert photo of PCB

How to connect to the Serial Port of this specific device:
Insert photo of PCB with markings for serial port

Serial connection parameters for Mellanox Spectrum SN2000 Series:

Parameter Setting
Baud rate 115200
Data bits 8
Stop bits 1
Parity None
Flow control None

COPY HERE THE BOOTLOG WITH THE ORIGINAL FIRMWARE


COPY HERE THE BOOTLOG ONCE OPENWRT IS INSTALLED AND RUNNING


Space for additional notes, links to forum threads or other resources.

  • ...
This website uses cookies. By using the website, you agree with storing cookies on your computer. Also you acknowledge that you have read and understand our Privacy Policy. If you do not agree leave the website.More information about cookies
  • Last modified: 2024/02/12 11:13
  • by 127.0.0.1