crush depth

MAC Address Conflicts Hurt

I've been running a server on my LAN here for a couple of years. This week, I built a new machine and am in the process of trying to migrate to the new hardware.

However, something odd happened... With an SSH connection open to the old server (cranberry), I booted the new server (vanilla). The open SSH connection began to sporadically and repeatedly hang and then come back to life. Shutting down vanilla caused the problem to go away. Some investigation with Wireshark showed odd TCP Retransmission events occurring, which I don't think I've ever seen before and certainly shouldn't be happening on a LAN.

Sometimes, SSH connections to vanilla would work fine, and the connections to cranberry would hang/resume instead. I couldn't determine any reason for this to be happening; all my hardware is in good shape.

Then, I tried to nmap vanilla and cranberry from my workstation...

$ nmap -vvv cranberry
...
MAC Address: 58:9C:FC:10:93:1F (FreeBSD Foundation)

$ nmap -vvv vanilla
...
MAC Address: 58:9C:FC:10:93:1F (FreeBSD Foundation)

I was firstly surprised at seeing a MAC address conflict, and then doubly surprised at seeing the string "FreeBSD Foundation" in what should be a hardware identifier.

Looking at ifconfig on cranberry and vanilla, I see:

$ ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 58:9c:fc:10:93:1f

$ ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 58:9c:fc:10:93:1f

I'd created bridge devices on both machines to which to attach TAP devices for bhyve virtual machines. Evidently the generated MAC address on both was the same, and this MAC address leaked out onto the LAN. Packets would essentially arbitrarily go to either machine if both happened to be switched on at the same time.

The manual page says:

  The if_bridge interface randomly chooses a link (MAC) address in
  the range reserved for locally administered addresses when it is
  created. This address is guaranteed to be unique only across all
  if_bridge interfaces on the local machine.  Thus you can
  theoretically have two bridges on the different machines with the
  same link addresses.  The address can be changed by assigning the
  desired link address using ifconfig(8).

Moral of the story: Assign your own MAC addresses to virtual devices. Pick an unused area in the MAC registry and go wild.