Facebook's Wedge: A novel approach to Top-of-Rack switches

Facebook and members of the Open Compute Project overhaul the critical Top-of-Rack (TOR) switch, with an eye on improving data center efficiency and reducing costs.

Figure A: See how using a TOR switch simplifies cabling to the rack.
Image: excITingIP.com

Facebook is not leaving one digital stone unturned in a quest to make its data centers as efficient as possible. Facebook engineers are looking everywhere they can in order to conserve energy, reduce costs, improve reliability, and simplify an immensely complex digital ecosystem.

The most recent device placed under the proverbial microscope is the Top-of-Rack (TOR) switch, which is an important component in current data-center design. The diagram in Figure A shows how using a TOR switch simplifies cabling to the rack.

While trying to wring out inefficiency, Facebook engineers were cognizant they had to work within the confines of the Open Compute Project (OCP). Frank Frankovsky, Chairman and President of the OCP, said that Najam Ahmad, Facebook's Director, Technical Operations, led the TOR switch project with participation from well-known networking organizations including Big Switch Networks, Broadcom, Cumulus Networks, Intel, and VMware.

Frankovsky specifically addressed the importance of having an open, disaggregated switch that will speed up innovation, help software-defined networking evolve, and provide flexibility to data centers that were concerned about scalability. Frankovsky added, "This is a new kind of undertaking for OCP — starting a project with just an idea and a clean sheet of paper, instead of building on an existing design."

The results: Wedge

The engineering team led by Ahmad came up with a novel idea when they designed Wedge, the codename for the new TOR switch. They decided to segregate device management from the switching technology, which allows more flexibility and the use of operating system-agnostic firmware. The new switch also incorporates the OCP micro server (PDF) shown in the exploded view in Figure B; it allows Wedge switches to employ a proven architecture called Group Hug. According to Facebook, Group Hug is:

A new common slot architecture for motherboards. This specification can be used to produce boards that are completely vendor-neutral and will last through multiple processor generations. The specification uses a simple PCIe x8 connector to link the SOCs to the board.

Figure B

Image: Facebook

This approach has merit. Anytime complexity can be removed, such as having servers, storage devices, and switches all use the same firmware, is a huge improvement.

That's the hardware piece of the puzzle; now let's look at the software developed for Wedge.

FBOSS: The Wedge's brains

Staying with the "clean sheet of paper" concept, the engineering team developed FBOSS, a Linux-based operating system designed specifically to:

  • Simplify the deployment of software solutions;
  • Boost network performance;
  • Provide metrics including cooling-fan behavior, internal temperatures, and voltage levels; and
  • Facilitate optimal power usage, performance, and predictive maintenance.

It may be a clean-sheet design for an OCP TOR switch, but the engineering team leveraged existing software libraries used to control "in the field" OCP servers, as seen in the flowchart in Figure C.

Figure C

Image: Facebook

In this Facebook Engineering blog post written by Yuval Bachar and Adam Simpkins, the team explains what makes FBOSS unique. (I searched but could not find whether FBOSS is an acronym or just a codename.) They write, "By controlling the programming of the switch hardware, we can implement our own forwarding software much faster. We also added a Thrift-based abstraction layer on top of the switch ASIC APIs, which will enable our engineers to treat 'Wedge' like any other service in Facebook. With FBOSS, all our infrastructure software engineers instantly become network engineers."

Another real advantage is FBOSS will give data-center engineers the ability to optimize network paths at the network edges — and not just a little. Bachar and Simpkins state, "We've managed to boost the utilization of edge network resources to more than 90 percent while serving Facebook traffic without a backlog of packets."

Last thought

When news of Wedge reached the tech media, it immediately garnered significant reaction. For example, Greg Ferro of Ethereal Mind said, "This shows that Network Operating Systems (NOS) like Cisco's IOS and Juniper's Junos are under attack. The destruction of the software moat means customers have more choices."

Ferro made a telling observation as well: "The arrival of independent NOS products has parallels with Windows/Linux. Some customers choose Windows at a high price, some customers choose commercial Linux, and others will choose free Linux."

It will be interesting to see how the proprietary vs. open source discussion spills into the network realm.

Automatically subscribe to TechRepublic's Data Center newsletter.

Also see

Disclaimer: TechRepublic and ZDNet are CBS Interactive properties.