Tuesday, December 14, 2010

Setup Link Aggregation for Fault tolerance and build fat network pipes from most network interfaces without any trunking software.

So anyway, one of the things I'd been wanting to do was figure out how to bond multiple network interfaces to give me some resilience on the net connection. Turns out it's pretty easy in Solaris using 'dladm' and ifconfig. Solaris 10 introduced the nemo framework for drivers and Solaris Nevada has more projects which build on said framework. In Update 2 of Solaris 10 support for data link aggregation was added.

From the manual for dladm

The dladm command is used to configure data-links. A config-ured data-link is represented in the system as a STREAMS DLPI (v2) interface which may be plumbed under protocol stacks such as TCP/IP. Each data-link relies on either a single network device or an aggregation of devices to send packets to or receive packets from a network.

The instructions here are correct for a server with at least 4 e1000 interfaces, where the first one was configured during the install as the network interface. You'll also want to do this from the console, because the first thing we're going to do is take down the network interface:

harrysolvm# pfexec ifconfig e1000g0 unplumb

harrysolvm# dladm show-dev

e1000g0         link: up        speed: 1000  Mbps       duplex: full e1000g1         link: up        speed: 1000  Mbps       duplex: full e1000g2         link: up        speed: 1000  Mbps       duplex: full e1000g3         link: up        speed: 1000  Mbps       duplex: full

Now we know what devices are available for aggregation. Lets use `dladm` to bond the two network interfaces together and bring the bonded interface back up.

harrysolvm# pfexec dladm create-aggr -d e1000g0 -d e1000g3 1

harrysolvm# pfexec dladm show-aggr 1

key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)            device       address                 speed           duplex  link    state            e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      standby            e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      standby

Agrregation completes and the link is in standby mode, next we need to plumb it.

harrysolvm# pfexec ifconfig aggr1 plumb

Now we give it an IP address:

harrysolvm# pfexec ifconfig aggr1 192.168.0.253 netmask 255.255.255.0 up

regular ifconfig to setup the link. Lets check the device state now.

harrysolvm# pfexec dladm show-aggr 1

key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)            device       address                 speed           duplex  link    state            e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached            e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached

We add some more nics to the device while the device is up and running.

harrysolvm# pfexec dladm add-aggr -d e1000g1 -d e1000g2 1

harrysolvm# pfexec dladm show-aggr 1

key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b0 (auto)            device       address                 speed           duplex  link    state            e1000g0      0:14:4f:1:c8:b0   1000  Mbps    full    up      attached            e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached            e1000g1      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached            e1000g2      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

Now lets show off and remove a nic from the link

harrysolvm# pfexec dladm remove-aggr -d e1000g0 1

harrysolvm# pfexec dladm show-aggr 1

key: 1 (0x0001) policy: L4      address: 0:14:4f:1:c8:b3 (auto)            device       address                 speed           duplex  link    state            e1000g3      0:14:4f:1:c8:b3   1000  Mbps    full    up      attached            e1000g1      0:14:4f:1:c8:b1   1000  Mbps    full    up      attached            e1000g2      0:14:4f:1:c8:b2   1000  Mbps    full    up      attached

There's one last thing you'll want to do:

harrysolvm# pfexec mv /etc/hostname.{e1000g0,aggr1}

The changes made with `ifconfig` don't automatically persist. It uses a combination of `/etc/hostname.`, `/etc/hosts` and `/etc/inet/netmasks` to infer the IP address of the interface. So you need to go putting the information in `/etc/network/interfaces` or some file in `/etc/sysconfig`.