Network Hardware Selection

Network Hardware Selection

February 6, 2017
Providers
Mikrotik, Ubiquiti

After several deployments of varying size and complexity, an offered opinion on the advantages and disadvantages of choosing Ubiquiti hardware for your next project. Originally written in early 2017, there are some updates from early 2018 relating to technical/ideological facets that have come up in the last couple years of operating Ubiquiti equipment. The article makes some assumptions:

  • You’re growing tired of that ddwrt or tomato router you’ve felt so cool operating for the last several years.
  • You’re considering segmentation as a requirement for the next deployment, necessitating managed switches
  • You’re wanting to use something more ’enterprisey’, maybe rack-mount-able hardware
  • You’re likely going to need multiple access points to cover the deployment area
  • You’re considering more things for your deployment, like surveillance or phones, and desire manufacturing conformity.

Context #

When first moving onward from a consumer grade router running ddwrt I jumped to pfsense on a supermicro 1U system that was both costly and loud. The big benefit though was pulling back the curtain on the open source software stacks that are relied on for the major network services that we all take for granted. This was a great learning experience in the major network services, but I was still using a consumer grade switch and didn’t have any notion of LAN segmentation. Secondarily, whatever I was learning/doing I wanted to deploy to the other physical nodes in my network and that meant committing family to costs that they wouldn’t take kindly to. I began searching for a manufacture that met some loose requirements:

  • Router equivalent needed to be sub $150 for deployment
  • Switch at 24 ports needed to be sub $300 for deployment
  • Access Points needed to be sub $100, and multi-unit deployable
  • Fan-less significantly preferred, as some of the deployments (my home) I sit very close to
  • Web UI that was reasonably featured
  • Terminal UI that was well documented
  • Concepts close to the IEEE 802 standards, rather than propietary niceties that would cause me to be locked in
  • Under the hood comfort, I had come from pfsense and wanted to have some level of transparency for the base OS
  • A semblance of a community to discuss ideas, pfsense had a really helpful one

I landed on Ubiquiti because their ERLite‑3 was less than $100, their ES‑24‑LITE was less than $250, and they offered more expensive products if I did decide to take on larger projects. At the time I had considered Mikrotik strongly, but there was a significant reliance on a Windows based controller software that I didn’t want to have at all. Ubiquti at the time had the UniFi line, but had not started branching out beyond simply Access Points. I was quite excited to interact with the community via the forums and IRC. They had some really great community managers which seemed to be able to respond to every inquiry. Their products were based on debian, the shell was familair, the web ui was useful. Overall it appeared to be an obvious match.

Ubiquti operates multiple product lines, but for networking they have two distinct lines:

EdgeMAX: which is marketed as their ‘broadband’ lineup, is clearly focused more on networking professionals that have experience in the professional grade, carrier class hardware. Originally had more options for hardware, but is Unifi is keeping hardware parity.

UniFi: which is marketed as their ’enterprise’ lineup, is clearly focused more on attracting those who wish to manage their network via a comprehensive interface. Most notably, this is their implementation of a fully software defined network.

With both you’re always going to end up dealing with Unifi because there is no analog in EdgeMAX for Access Points. Generally features have showed up within EdgeMAX first and then trickled down to UniFi. This may not be the case in the future as it’s been widely discussed about how Ubiquiti has expanded the developmental effort for UniFi. While, at the same time, some of the most prolific contributors to EdgeMAX were departing. There has been attempts to loosely re-assure the community, but my personal pulse on the matter is that EdgeMAX is now a second priority. I’d almost rather see EdgeMAX loose its web UI entirely, and just be a testing grounds for the new features/specifications that get wrapped in the UniFi web UI.

I’ve deployed both systems several times of varying sizes. I use Edgemax at home, and with my family, due to familiarity and wanting to track to new features. I use UniFi deployments for small offices, and for friends/colleagues who don’t want to muck with the complexity of the EdgeMAX interface. With EdgeMAX you can reach under the hood at two levels, the first is the EdgeOS terminal interface that is based on Vyatta, second level is root shell on their instance of Debian. With UniFi, you best stick to the controller software as point deviations are really not supported well and get frequently overwritten. If someone has the intention of digging deep into their networking infrastructure, and we’re talking really deep, then EdgeMAX likely makes sense for them. Anyone who inquires with me about this I will tell to use UniFi, because it covers the vast majority of use cases.

The Good

  • Ubiquiti stuff is really inexpensive
  • Ubiquiti are deploying new hardware quite often, recently 10G switching
  • There is a pretty large community of people deploying the systems, practitioners are quite accessible through both the forums and IRC
  • EdgeOS is a fork of Vyatta, which was as a convention is similar to other enterprise class interfaces, as well as tracks closely with the concepts within the IEEE 802 standards

The Bad

  • Ubiquti has very little transparency, literally none of their product lines have any published road maps
  • Ubiquiti seems to ravenously chase an objective, then loose interest (remember mFI?)
  • Ubiqitui seems to be significanly preocupied with UniFi as a product line, EdgeMAX is largely silent
  • Ubiquiti has started moving towards cloud deployments of UniFi product, signaling that they’ve got the architecture to make future features deploy behind a subscription model
  • Even though people may say that its Debian under the hood, its Ubiqiti’s build of Debian, you’re not gonna want to muck about with it too much.
  • The community is largely filled with new/novice people who don’t want to learn fundamentals, and Ubiquti staff don’t tend the forums well enough to keep signal:noise reasonable

Alternatives to UBNT #

Added in 2021: Just use Mikrotik.

I’m a huge fan of pfsense and ran it for quite a while. I jumped ship primarily because I had to purchase relatively expensive hardware (around $400 at the time, before the proliferation of ARM devices) just to deploy the router. Since then they’ve come up with their own line of hardware that is quite impressive. The major issue I have is that I’d really like to have homogeneous manufacture for a network deployment, and there doesn’t appear to be a pfsense analog for switching. You’d have to adopt some switch manufacture, and then why not consider Ubiquiti, or similarly Mikrotik.

There was some recent drama related to discussions about pfsense being an open source project. There is another project called opensense that forked and started really modernizing the interface, it’s worth looking at.

Lot’s of Ubiquiti adopters like to scoff at ’tik’ people, but the company has both a compelling portfolio of hardware as well as software. To me, there is a bit more desire to consider Mikrotik as a company due to their organization, as well as the internalization of manufacturing. The appear, at initial glance, to be the networking hardware equivalent of purchasing from a community. When I initially adopted Ubiquiti products there was a distinct lack of web control for the RouterOS lineup. It required a Windows based client, which was a non-starter for me. Since then their control software has significantly improved, during the same time Ubiquti realized their entire Unifi line of products. I’m not personally well versed in Mikrotik to make a current state comparison, but I’d imagine the momentum of Ubiquti to be a bit greater on this front.

Netgear is active in this space, Dlink as well, I ignore both of them due to their approach on consumer routers being so closed. We’ve seen rampant botnets forming from the gaping holes or back-doors left in consumer devices, these communities who want to re-claim their hardware should be embraced by these manufactures.

With Cisco/Belkin changing hands of Linksys I was skeptical about what their products would become. Turns out they embraced the community, and have even stuck up on the FCC front for modification of router firmware. There just isn’t scale-ability for routers, or any managed switches, in their portfolio.

I’ll very likely continue to deploy Ubiquti products, despite the downsides, because there isn’t a clear homogeneous solution for home-to-large-scale deployments that has some level of openness and reasonable costs. I am always considering alternatives, with a hope that there emerges a complimentary community to pfsense for switching.

For larger scale deployments I’ve started moving in to cumulus with their incredibly cost effective L2 RMP switch which is roughly double the cost of an ES-48-Lite with a similar feature set. Most deployment costs are driven by the edge layer of the network, so a low cost capable device on the edge is important. The major upside with cumulus, other than Linux, enterprise class software, responsible support… is that there is 10/40/100G options that actually scale compared to the ES-16-XG platform. For example you’d spend 35% the cost deploying 48 ports of SFP+ with the ES-16-XG platform compared to the cumulus alternative, but you’d be significantly limited on your actual L3 capabilities.

Learned Ubiquiti defaults that bite (Added in 2018 Q1) #

We’ve plunged into building out a moderately large laboratory network using Ubiquiti’s ES-16-XG and ES-48-Lite. By default these switches have some defaults that can cause many problems.

Auto-negotiation is not to be trusted #

Most people will say only to use auto-negotiation for provisioning and don’t rely on it for production, they are right. Make sure to have a USB-to-Serial cable on you because there is a whole class of network interfaces that won’t work when you plug them into the ES-16-XG copper interfaces.

LACP isn’t a default #

The first is when you set up Link Aggregation the default is to use the older static protocol instead of the more modern 802.3ad LACP standard. You’ll have to set each link to use LACP with this parameter:

configure
interface 3/1
no port-channel static

If you do not do this, you’ll see periodic drops between switches that don’t make a lot of sense. This can be very frustrating to troubleshoot because you’ll sometimes get a race condition where both switches just agree to work over the static aggregate, then at a later time you drop. Always use LACP between switches.

Updated, as of v1.8.0 LACP is the default

STP is a default, but you better have a root bridge #

Once you start using LACP you’ll see that STP is enabled by default. If you do not have a root bridge specified in your network then STP will adjudicate by selecting the oldest MAC in your network… yeah… So, assuming you’re creating a “simple” network and trusting STP you will want to select a device to be the root bridge, this likely will be the device that is “one armed” to your router. On that switch you will want to:

configure
spanning-tree mst priority 0 0

I had created a pretty bad mess when a very old ES-48 was behaving as the root bridge of our network, while functionally it was a leaf unit.

Lookback (Added in 2018 Q1) #

This article was written originally in middle 2017. At the time there was some awareness of a really bad wire fraud incident within Ubiquiti. This shook the community a little bit, but by and large Ubiquiti has a strong hold on the “homelab” market because of their low cost… so skepticism is weighed heavily against the thought you might have to spend $1-2,000 moving ideologically away from a manufacture.

A notorious short-seller came out in 2017 with a long list of partial truths to encourage people to dump Ubiquiti stocks. The problem here is that he had enough partial truths put together in a list that people who cared about the companies organizational health (not necessarily financial return) should have been pretty concerned.

During the same year there was an advisory about deploying Ubiquti products. The vulnerabilities could be side stepped through doing some more rigorous deployment, however if you examine the contact timeline it feels very similar to how Ubiquiti treats their “community”.

If you spend any time in the “community” you will quickly notice that the majority of responses are from participants instead of Ubiquiti employees. A couple major employee respondrs, like ancheng left the company under some strange circumstances that seemed like an internal quarrel with senior leadership. Ubiquiti has tried to “right” this out by hiring cmb, notorious from the pfsense project, and benpin who has a ton of helpful videos. Even with staffing up… the “community” is hit and miss, and Ubiquiti likes to claim it as a primary form of support for their products.

Most recently there have been some SEC Subpoenas related to internal financials. All in all, Ubiquiti does not seem like a healthy company for employees or investors. So when you’re purchasing these products you should consider that the point release of software is the most polished version you’re going to get. Don’t purchase products for deployment based on their potential, many saw what happened when banking hard on the ES-16-XG which has had myriad problems (and has had major features ripped from the firmware releases(stacking)).