This is an old revision of the document!
Mellanox Spectrum SN2000 Series
The Mellanox Spectrum SN2000 Series Switches are Managed Ethernet Switches with a maximum speed of 100Gb/s and up to 56 ports.
They use the switchdev API to offload the kernel's forwarding plane to its ASICs, allowing for more network throughput.
More info: https://github.com/Mellanox/mlxsw/wiki
Supported Versions
Hardware Highlights
Installation
Unfortunately, there is currently no pre-built OpenWrt image for this target in the Firmware Selector. There are two options to obtain an image and set up a working OpenWrt system on this device:
- You can build your image manually following the instructions described here.
Flashing the image is done as you do it with any other x86 target: https://openwrt.org/docs/guide-user/installation/openwrt_x86#installation.
The instructions on this page will show you how to install OpenWrt on the switch by flashing it from a USB device. You have to connect the host PC to the serial port of the network switch to navigate through the BIOS and the installation steps. The parameters of the serial connection can be found here.
This will erase the complete disk and flash OpenWrt, removing existing operating systems, including the commonly pre-installed ONIE or Onyx.
Preparing the USB device
The USB device will serve as a kind of Live USB. If you already have a Live USB from another Linux distribution, you can also use that and skip to the next step.
First, plug the USB device into your PC, identify your USB device (e.g., by running fdisk -l; the instructions here will use /dev/sdx), and flash the OpenWrt image to it.
This will erase all your data on the USB device.
gunzip -k openwrt-x86-64-generic-ext4-combined-efi.img.gz # Replace sdx with your device dd if=openwrt-x86-64-generic-ext4-combined-efi.img of=/dev/sdx
If you are using an x86_64 image with a default root partition size of 104MiB, we must enlarge it and its filesystem. Some Linux distributions automatically mount the root partition, so you may have to unmount it before.
parted -s /dev/sdx resizepart 2 100% resize2fs -f /dev/sdx2
Copying and flashing the image
Mount the root partition of the USB device and copy the OpenWrt image on it.
mkdir /mnt/usb # This is for OpenWrt. If you used another distributions Live USB, adjust the paths accordingly. mount /dev/sdx2 /mnt/usb/ cp openwrt-x86-64-generic-ext4-combined-efi.img.gz /mnt/usb/root/ umount /dev/sdx2 rmdir /mnt/usb
Next, (safely) unplug the USB device, plug it into the Mellanox switch, and connect your PC to its serial port.
To enter the BIOS, reboot the switch and press CTRL+B while you see the BIOS information text (American Megatrends …).
The default password to enter the BIOS is admin.
Switch to the Save & Exit tab and select your USB device from the Boot Override options.
When booted, press Enter on your keyboard to activate the console and start flashing the image.
cd ~ gunzip openwrt-x86-64-generic-ext4-combined-efi.img.gz # The SSD should be '/dev/sda', verify before flashing! dd if=openwrt-x86-64-generic-ext4-combined-efi.img of=/dev/sda
Finally, if the flashing process is successful, reboot, and the switch should boot into the newly installed OpenWrt system.
As mentioned, the root partition size might be only 104 MiB if you used a default x86_64 image (configuration). To increase it and the filesystem, please follow the steps of the following link: https://openwrt.org/docs/guide-user/installation/openwrt_x86#expanding_root_partition_and_filesystem.
If you haven't used a manually built image for the installation, you must proceed with the following section to install all required packages.
Install required packages
The following steps are needed if you haven't built your image manually and used the generic x86_64 snapshot image from the Firmware Selector. You must also repeat those steps if you do a sysupgrade on the device.
First, plug in an ethernet cable into the management port and make sure that it has internet access. By default, the port has the IPv4 address 192.168.1.1/24, but you can change it by following the steps provided here: https://openwrt.org/docs/guide-user/network/network_configuration.
After you have gained internet access, you can install the required package through the opkg package manager. Before you can do that, update the list of available packages:
opkg update
After that, you can install the packages required for the QSFP ports, LEDs, and hardware watchdog to work:
opkg install mlxsw_spectrum-firmware kmod-mlxfw kmod-mlxsw-core kmod-mlxsw-i2c kmod-mlxsw-minimal kmod-mlxsw-pci kmod-mlxsw-spectrum kmod-mlx_wdt kmod-leds-mlxcpld
If you also want to monitor the temperature of the CPU, RAM, and SSD drive, install the following additional packages:
opkg install kmod-hwmon-coretemp kmod-hwmon-drivetemp kmod-hwmon-jc42 kmod-i2c-i801 kmod-mlxreg
If you have a Mellanox Spectrum SN2201 or SN4800, I also recommend the following package for hardware monitoring:
# For SN2201 opkg install kmod-mlxreg-sn2201 # For SN4800 opkg install kmod-mlxreg-lc
Finally, reboot the device, and the QSFP ports of the switch should be functional now.
Image Configuration
If you want to build your image manually for this device, follow the steps provided on this page: https://openwrt.org/docs/guide-developer/toolchain/use-buildsystem.
When you are inside the build system configuration interface (make menuconfig), select the following target configuration:
- Target System:
x86 - Subtarget:
x86_64 - Target Profile:
Generic x86/64
It isn't mandatory, but you should also unselect Use Console Terminal (in addition to Serial) under the Target Images section to avoid possible problems while using the serial console.
I also recommend increasing the Root filesystem partition size (in MiB) (e.g., 512).
For the QSFP ports, LEDs, and hardware watchdog to work, you have to select the following packages:
- Firmware:
mlxsw_spectrum-firmware - Kernel modules → Network Devices:
kmod-mlxfw,kmod-mlxsw-core,kmod-mlxsw-i2c,kmod-mlxsw-minimal,kmod-mlxsw-pci,kmod-mlxsw-spectrum - Kernel modules → LED modules:
kmod-leds-mlxcpld - Kernel modules → Other modules:
kmod-mlx_wdt
If you also want to monitor the temperature of the CPU, RAM, and SSD drive, you can select the following kernel modules:
- Kernel modules → Hardware Monitoring Support:
kmod-hwmon-coretemp,kmod-hwmon-drivetemp,kmod-hwmon-jc42 - Kernel modules → I2C support:
kmod-i2c-i801 - Kernel modules → Other modules:
kmod-mlxreg
If you have a Mellanox Spectrum SN2201 or SN4800, I also recommend the following package for hardware monitoring:
- Kernel modules → Other modules:
kmod-mlxreg-sn2201(SN2201),kmod-mlxreg-lc(SN4800)
You can also append the following configuration symbols to the .config file after you have selected the generic x86_64 target inside make make menuconfig:
CONFIG_PACKAGE_mlxsw_spectrum-firmware=y CONFIG_PACKAGE_kmod-mlxfw=y CONFIG_PACKAGE_kmod-mlxsw-core=y CONFIG_PACKAGE_kmod-mlxsw-i2c=y CONFIG_PACKAGE_kmod-mlxsw-minimal=y CONFIG_PACKAGE_kmod-mlxsw-pci=y CONFIG_PACKAGE_kmod-mlxsw-spectrum=y CONFIG_PACKAGE_kmod-leds-mlxcpld=y CONFIG_PACKAGE_kmod-mlx_wdt=y # Hardware Monitoring CONFIG_PACKAGE_kmod-hwmon-coretemp=y CONFIG_PACKAGE_kmod-hwmon-drivetemp=y CONFIG_PACKAGE_kmod-hwmon-jc42=y CONFIG_PACKAGE_kmod-i2c-i801=y CONFIG_PACKAGE_kmod-mlxreg=y # For SN2201 CONFIG_PACKAGE_kmod-mlxreg-sn2201=y # For SN4800 CONFIG_PACKAGE_kmod-mlxreg-lc=y # Optional, but recommended CONFIG_GRUB_CONSOLE=n CONFIG_TARGET_ROOTFS_PARTSIZE=512
After the build process (make) has finished, you can find the compiled image at bin/targets/x86/64/openwrt-x86-64-generic-ext4-combined-efi.img.gz.
Upgrading OpenWrt
The instructions here use 192.168.1.1 as the IPv4 address of the switch. Replace it with your configured IP address where applicable.
LuCI Web Upgrade Process
- Browse to
http://192.168.1.1/cgi-bin/luci/mini/system/upgrade/LuCI Upgrade URL - Upload your manually built or the generic x86_64 snapshot image from the Firmware Selector for sysupgrade to LuCI
- Wait for reboot
Terminal Upgrade Process
It is important that you put the firmware image into the ramdisk (/tmp) before you start flashing.
sysupgrade
If you have built your image manually, transfer the image from your host machine to the switch first:
scp -O openwrt-x86-64-generic-ext4-combined-efi.img.gz root@192.168.1.1:/tmp
Then, log in via SSH and enter the following command:
sysupgrade /tmp/openwrt-x86-64-generic-ext4-combined-efi.img.gz
If you want to use the generic x86_64 snapshot image from the Firmware Selector, log in via SSH and enter the following commands.
But remember that the QSFP ports won't work anymore after the switch has rebooted, so you must repeat the steps described here after the
sysupgrade has finished.
cd /tmp wget https://downloads.openwrt.org/snapshots/targets/x86/64/openwrt-x86-64-generic-ext4-combined-efi.img.gz sysupgrade /tmp/openwrt-x86-64-generic-ext4-combined.img.gz
Basic configuration
Network interfaces
The default network configuration is:
| Interface Name | Description | Default configuration |
|---|---|---|
| br-lan | Management Port | 192.168.1.1/24 |
| lo | Loopback | 127.0.0.1 |
Buttons
The Mellanox Spectrum SN2000 Series has the following buttons:
| BUTTON | Event |
|---|---|
| Reset | reset |
Hardware
Info
Photos
Front:
Insert photo of front of the casing
Back:
Insert photo of back of the casing
Backside label:
Insert photo of backside label
Opening the case
Note: This will void your warranty!
Describe what needs to be done to open the device, e.g. remove rubber feet, adhesive labels, screws, ...
- To remove the cover and open the device, do a/b/c
Main PCB:
Insert photo of PCB
Serial
How to connect to the Serial Port of this specific device:
Insert photo of PCB with markings for serial port
Serial connection parameters for Mellanox Spectrum SN2000 Series:
| Parameter | Setting |
|---|---|
| Baud rate | 115200 |
| Data bits | 8 |
| Stop bits | 1 |
| Parity | None |
| Flow control | None |
Bootlogs
OEM bootlog
COPY HERE THE BOOTLOG WITH THE ORIGINAL FIRMWARE
OpenWrt bootlog
COPY HERE THE BOOTLOG ONCE OPENWRT IS INSTALLED AND RUNNING
Notes
Space for additional notes, links to forum threads or other resources.
- ...
