Networking

Short Version

Whenever you set up a system with more than one Ethernet connection, make sure the traffic is going where you think it’s going, or else it could be going everywhere!

Dual-homed device Issues

An issue that we’ve seen come up across the campus network more than once has to do with how dual-homed (a system that has more than one active physical connection to the network) systems interact with the properties of Ethernet. In many cases, the reason for dual-homing a system is to provide redundancy or some level of load balancing. The IP addresses used for those multiple network interfaces could be in the same subnet or could be in different subnets. The important issue is whether or not the network “hears” from each of those network interface cards (NICs).

Ethernet Switching Explanation

The way Ethernet switching works is quite simple, at least if you have just one NIC in your system. Your device sends out an Ethernet packet, either initiated by an application on your system or in response to a packet you’ve received from another host. The network switch to which your device is connected hears and learns the hardware address of your Ethernet NIC (referred to as the MAC, or Media Access Control, address), as does every other Ethernet switch in the path between you and either the destination of your reply or the nearest router. Each of those switches in that path maintains a Source Address Table, where it keeps the information on which port it hears a MAC address for up to five minutes. Obviously, if a device stays pretty chatty, it’s going to stay in the Source Address Table for a long time. When an Ethernet switch receives a packet from someone on the network destined for your MAC address, the switch knows immediately which of its ports to send that packet out.

Now, here’s the gotcha: if a device’s NIC NEVER SPEAKS on the network, or hasn’t spoken in more than five minutes, and a switch receives a packet destined for that MAC address, the rules of Ethernet require the switch to send that packet out EVERY switch port in that destination VLAN. If no switch on the network has heard any traffic from that MAC address in over five minutes, the packet(s) to that address will be flooded out EVERY single switch port configured for that destination VLAN across the entire campus network until the network “sees” a packet from that NIC. With a one-NIC device that’s not broken, once it receives that packet and replies to it, then all of those switches will learn again where that address is and no further packets will be flooded.

But, what if that NIC never sends a reply? Or what if it’s a dual-homed device that either intentionally or unintentionally is set to have all outbound traffic go out just one of the two NICs in that system? On many systems, when two NICs are installed and configured to be in the same subnet or VLAN, one of those NICs is chosen as the “default route” for all traffic leaving the system. You may have traffic coming in to both NICs as designed, but all of the traffic may be leaving from the one NIC and therefore with just one source MAC address (even though the packets may be from both IP addresses). When that happens, and no traffic is actually sourced from one of those two NICs, traffic destined for the “non-default” MAC address could flood to every switch port in that VLAN, as described above.

Solution

The key is to make sure, if you feel you need to have a dual-NIC configuration, that you are actually sending traffic out both interfaces. One way to check this is to use a packet capture tool like Wireshark and check both interfaces for traffic from their respective MAC addresses. If you find that all of the outbound traffic is only going out from the one NIC (the “default route”), you might consider creating a static route for a specific host or subnet that points to the other NIC, just to make sure the network sees both MAC addresses.