SWY's technical notes

Relevant mostly to OS X admins

802.3ad (LACP) Bonding for fun

I happened to have a 15″ Retina, 2 Thunderbolt<->Ethernet adapters, a Synology DS1812+ NAS, a stack of 3 Dell 5548P switches, few extra minutes, and some curiosity at my disposal today, so I decided to see what sort of real-world numbers I could push between these 2 devices over 2 bonded Ethernet connections.

First up: the MBP. To create a Virtual Bonded Interface, start by plugging in the Thunderbolt-Ethernet adapters- the OS will need to see the interfaces present before they can be bonded.  Then, from the Network System Preferences, click on the [+] at the bottom, and select Manage Virtual Interfaces:

Screen Shot 2013-02-14 at 5.16.25 PM

In the new window, click [+] again, and select New Link Aggregate…

Screen Shot 2013-02-14 at 5.16.38 PM

Select the Ethernet interfaces to bond, and give them a name.

Screen Shot 2013-02-14 at 5.16.56 PM

Before these are connected to the network, you need to configure a pair of switch ports to also participate in the bond. In my Dells, that’s configured via Switching: Link Aggregation: LAG Membership, where LAG stands for Link Aggregation Group. Enter the Edit mode (vs the Summary they start in), and indicate which ports will be in your currently indicated LAG. Click first in the LAG row to bring them into the group, then the LACP row above it.  After saving the changes, you should be able to connect both Ethernet connections to your specified switch ports, and bring up a bonded connection.

LAG MembershipThis bonded connection isn’t too useful on its own without another bonded device. Next step is to configure LAG2 for the NAS in the same way.

Finally, we configure the NAS to do 802.3ad. In the Synology, that’s configured via Control Panel: Network: Network Interface: Create. Choose the default 802.3ad link aggregation, both interfaces, any VLAN options, and apply.  A couple of pings will drop, but the new bonded connection will be set up. Attach both Ethernet cables to the LAG2 configured ports, and it should be back on your network, with twice the data path.

To measure the value of LACP, I tested some Finder file copies to and from the NAS. I picked a 12 Gig folder of .DMG files- many large files, as to have minimal file creation overhead. Naturally, it helps to have fast disks at both ends- the MBP’s SSD can keep up far faster than this, and it looks like the NAS configuration of 8 drives with dual parity can also saturate 2 GigE links.

Screen Shot 2013-02-15 at 2.31.42 PM

The writes aren’t quite as steady as I’d hoped for, but holding in the 180+MB/sec range for network writes from a laptop is pretty nifty. There’s also the benefit of redundancy- one side of the bond can be lost (cable out, switch down (presuming the LAG spans multiple switches)), and the network connection isn’t broken. And, if multiple, non-LAG clients are addressing the NAS, it does a great job of providing high performance to both clients.

The price you pay in this configuration is in flexibility- you can’t just plug in one of these 2 adapters to an arbitrary network port like you can out of the box- those 2 Tbolt<->GigE adapters are now part of that bond, not separate devices to be used as single links. You’ll need a 3rd adapter if you wanted the ability to have both the high performance bond AND a standard, not-bound-to-the-LACP-configured-ports Ethernet connection- perhaps a “at my desk” vs “anywhere else” use case.

 

Addendum1 20160304:

I’ve realized my testing methods above are flawed.  I only used the data from Activity Monitor to answer the question “how fast are we transferring data?”, the answer above is 194MB/sec.

This value is not a measure of true throughput.  I revisited this topic today, timing how long a specific file transfer took with 2 bonded interfaces and an LACP configuration on my switch, vs how long it took using a single port, no bonding.  Answer: exactly the same file transfer speeds.  Despite Activity Monitor (now on 10.11.3) still reporting 2x the transfer rate under the bond, the actual time to transfer a 1.5GB .dmg was unchanged by bonding ports.  Sorry to say that despite the visual feedback in Activity Monitor, port bonding cannot double your transfer rates from a single source.

So what is it good for?  Maximizing performance from a server to multiple clients.  In this case, if you’re using a MBP as a server… well, more power to ya.

Advertisements

7 responses to “802.3ad (LACP) Bonding for fun

  1. Brian McGroarty September 22, 2013 at 12:12 am

    I’ve done similar with my iMac’s built in ethernet adapter and a single Thunderbolt adapter. In case this saves anyone a bunch of hair pulling: One catch is that if you do this, you’ll want to ensure that your built-in ethernet adapter is the first interface in the new link group. The link group inherits the MAC address of the first adapter, and the other adapters effectively change. If your primary adapter’s MAC address changes, App Store purchases may mysteriously refuse to run as some check the primary ethernet adapter’s MAC address as part of the purchase receipt validation.

    That said, I saw network utilization numbers pretty similar to what the author encountered. I used a Netgear GS108T-NAS and an Apple Thunderbolt to Ethernet adapter. The two together were only about $115 US, shipped from Amazon.

    • syuroff September 22, 2013 at 6:11 am

      Nice tip about the adapter order, Brian. That’s a detail I didn’t dig into, but could certainly lead to some unwanted results if not followed.

  2. glenn May 8, 2014 at 8:50 pm

    can it be done with 2 usb to gigE adapters? I have had no luck.

    • syuroff May 8, 2014 at 9:13 pm

      Untested by me, but they should be legit ethernet adapters, capable of bonding.

      Note that if you’re using the Apple adapters, they don’t do GigE, only 100B. Not much point in that…

  3. Avery January 20, 2016 at 8:47 pm

    Hi syuroff — thanks so much for the post! I was trying this for my late 2013 MBP OSX 10.10, with a Netgear GS724Tv3 smart managed switch and a QNAP TS-653pro NAS. I configured the LAG on the switch for both the NAS and MBP as LACP. The NAS is setup with two NICs 802.3ad/LCAP bonded, and the MBP with the Link Aggregate setup and showing as ‘green’ for two thunderbolt to GigE adapters. Using two laptops, I can definitely pull > 125MB/s from the NAS, so I know both NICs are transmitting concurrently. I’m unable to pull > 110MB/s from the NAS solely from the MBP… seems to still be limited to a single link.

    I’ve also read a couple references to the fact that for a single file you can’t use LACP to double bandwidth… only for multiple requests concurrently. Furthermore, is sounds like setting up MPIO is the only real way to achieve > 1 Gb connection. Given I can’t demonstrate that I can (even with files in tandem) pull > 110MB/s, do you have any thoughts on this? Clearly you pulled more across your LAG, and I’m unclear if one folder copy with many files acts as a ‘single request’ or if it acts as ‘multiple parallel requests’. Any thoughts or insight you may have would be great to hear. Thanks!

    • swy January 20, 2016 at 9:16 pm

      I think I need to do some copies and calculate my own MB/sec values with a stopwatch vs trusting what Activity Monitor shows- since this original post, I too have read documentation that 2 LACP combined devices aren’t going to get 2x speed: so either I’ve disproven that, or Activity Monitor is deceiving us. My observations above are based on the network speed graph, where I could consistently copy files with that type of results. I’m now curious what will happen if I start pulling around 200MB/sec and then pull a link. If it’s really using both ports, the copy should continue, but drop to 100MB/sec neighborhood.

    • macseefoo December 2, 2016 at 8:01 am

      Hi lads, nice post! just saw his and can concur with all your findings. We’ve implemented LACP VLANs s a zyxel gs2210-24 L2 switch, with 4 hosts [3 x macmini machines and old MACRO 4,1] all at  OSX 10.11.6 with link aggregation, JUMBO frames (9K), several non-sticking sysctl net.inet.tcpxx changes.

      Hardware: using OWC Thunderbolt Bolt docks and a server USB3 with realtek RT8351 Ethernet GbE adapters etc on macminis, .Up to 3 x GbE in a LAG for each host. These are all ganged up as bonded Link Aggregations on the MAmcnis and the two ethernets EN0 & En1 on the MacPro 4,1 (x2 lanes)

      Static PORT based VLANS in the gs2210 to make the Link Aggregation groups. .. basic stuff.

      So, whats happens…? OSX tested with incompressible PRORES footage via SCP, rsync , AJA System Test network files systems [in AFP and SMB3] (default in OSX 10.11.6) source and targets are SAS and TBOLT based 700MB/sec disk arrays. The source BOND was network index 0 (the first one0.

      Our results were a less optimistic 110MB/sec (Write)and 106MB/sec Read on source host. Same in reverse. The data rate is an average. for 300GB+ the data rate is not sustained.a as memory buyers get full it seems (netstat -mm)

      again metrics from:
      scp command, rsync –progress –stats and AJA system test )the latter is a little over optimistic for network mounts)….

      and.. SMB3 was simply dreadful.. slower… beats me.. We use AFP where we can for network mounted file systems .. best results here as well as rsync from-file.mov me@lacp04:/Volumes/array03/xx and scp. etc

      HOWEVER! the PARALLEL data rate is sustained when several 3-4 concurrent data transfers is undertaken. SO concurrent workflow from a host to server benefits most from LACP. Very promising!

      As realised, this is the way LACP works for at least host and all between Switches to provide a much fatter pipe.

      Its reasonable to use for SAS based LTO4 tape backup (BRU-PE and HP 1840) at that rate without stopping the tape drive to archive/backup over LACP LAN .. so thats cool.

      Thus LACP so far does not accelerate a single session.. this is some marketing myth by some NAS vendors me thinks.

      DPR (Dynamic path reconnect) and MIOP sadly needs some fabric switching hardware.. need to mortgage the house …. in a host buss adapter 9 HBA) or aa fanciful “super smart NIC/ethernet switch”…. do utilise ALL the paths for each separate packet. Sadly LACP doesn’t this.

      MORE: looks like LACP work great with a iSCSI for a “direct attached” NAS… looking at this at present as well. Lots more flexible and cost effective that expensive close 48TB Thunderbolt disk arrays..:)

      macseefoo,
      Hong Kong

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: