RouteBricks: Code

How to build a RouteBricks server

Note: The following instructions assume that the servers have a copy of CentOS 5.3 64 bit installed. The OS image is available from http://www.centos.org. The RouteBricks code and instructions have not been tested on any other Linux distribution.

Install gcc

# yum install gcc.x86_64 gcc-c++.x86_64

Disable unused services

# ntsysv

RouteBricks requires only syslog, sshd, network and crond. All others are optional and if turned on may negatively affect server throughput performance.

Download 2.6.24 kernel source code:

# cd /usr/src
# wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.24.7.tar.bz2
# tar xvjf linux-2.6.24.7.tar.bz2
# ln -s linux-2.6.24.7 linux

Download the kernel config file for RouteBricks:

# wget http://routebricks.org/code/rb.config
# cp rb.config linux/.config

Download the 1Gb driver for the new kernel

# wget http://downloads.sourceforge.net/project/e1000/igb%20stable/1.3.28.4/igb-1.3.28.4.tar.gz

Compile and install the 2.6.24.7 kernel

# cd linux
# make -j 8
# make -j 8 modules_install
# make -j 8 install

Compile and install the 1Gb driver for Intel on-board NICs

# cd ..
# tar xvzf igb-1.3.28.4.tar.gz
# cd igb-1.3.2.8.4
# cd src
# make
# make install

Make sure /boot/grub/grub.conf points to the new kernel 2.6.24.7 and reboot.

Note: The network interface may not load by default after reboot. If not, from the shell one has to manually install the driver module:

# modprobe igb

Download and unpack Click runtime

# cd /usr/src
# wget http://read.cs.ucla.edu/click/click-1.7.0rc1.tar.gz
# tar xvzf click-1.7.0rc1.tar.gz
# ln -s click-1.7.0rc1 click

Patch the Linux kernel for Click runtime

# cd linux
# patch -p1 -b < /usr/src/click/etc/linux-2.6.24.7-patch

Download the RouteBricks 10Gbps driver patch

# cd /usr/src
# wget http://routebricks.org/code/ixgbe.netdevice.h
# cp ixgbe.netdevice.h /usr/src/linux/include/linux/netdevice.h

Compile the new kernel and the igb driver

# cd linux
# make -j 8
# make -j 8 modules_install
# make -j 8 install

Re-install the Intel on-board NIC driver

# cd /usr/src/igb-1.3.28.4
# cd src
# make install
# reboot

Configure the Click runtime

# cd /usr/src/click
# ./configure --enable-linuxmodule --enable-warp9 --enable-multithread=16 --disable-userlevel

Compile and install the elements to support multiple RX/TX queues

# cd /usr/src
# wget http://routebricks.org/code/mq-click-1.0.tgz
# tar -C /usr/src/click/elements/linuxmodule -xvzf mq-click-1.0.tgz
# cd click
# make elemlist
# make install

Download Intel 10 Gbps NIC driver source code

# cd /usr/src
# wget http://downloads.sourceforge.net/project/e1000/ixgbe%20stable/1.3.56.5/ixgbe-1.3.56.5.tar.gz
# tar xvzf ixgbe-1.3.56.5.tar.gz

Download and apply driver patches (they only work with Intel 'Oplin' 82598EB NICs)

# wget http://routebricks.org/code/ixgbe-1.3.56.5.patch
# cd ixgbe-1.3.56.5
# patch -p1 -b < /usr/src/ixgbe-1.3.56.5.patch

Install driver with patches and reboot

# cd src
# make CFLAGS_EXTRA="-DIXGBE_NO_LRO -DCLICK_ENABLED" install
# reboot

Validating the server setup with a minimal forwarding test

The following instructions allow to verify that the server setup is correct. They assume a server that can run at least 8 kernel threads (e.g., 4 cores with SMT or 8 cores without SMT). The NIC must be a dual port Intel 10Gbps 82598EB ('Oplin').

Start the driver enabling RSS with 8 RX and 8 TX queues.

# rmmod ixgbe # modprobe ixgbe RSS=4,4,4,4

Start the Click kernel module with the minimal forwarding configuration

# cd /root
# wget http://routebricks.org/code/minfwdtest.click
# click-install -t 8 minfwdtest.click

Once the Click threads are running, all traffic received on port eth0 will be forwarded (independently of the content) to eth1 and viceversa. (the eth0 and eth1 interface names must be assigned to the 10Gbps NIC).

How to build a RouteBricks router

Unfortunately, the Click configuration to instruct a cluster of servers to act as one single router is not available public domain yet.