SolutionBase: PCI Express brings a new bus standard to workstations and servers

The next time you crack the case on a new workstation, you may find yourself looking at a different kind of expansion slot—PCI Express. Here's what's involved in this new bus standard.

The more things change, the more things stay the same. When it came to adding cards to a system, first came the Industry Standard Architecture (ISA) expansion bus, followed by the expanded ISA bus (EISA), which was supplanted by the Peripheral Component Interconnect bus. The battle for the successor to PCI has been going on for some time, and like EISA vs. ISA, many of the names are similar. Now a new kid is on the block looking to supplant them all: PCI Express.

As the emerging standard for expansion boards, PCI Express slots will start appearing in new systems that you deploy. Understanding what's coming in PCI Express is going to be more important over time. Let’s look at the retiring standards, the competitors for the throne, and the new champion.

PCI Peripheral Component Interconnect Bus

The desktop variant of PCI is a 32-bit bus operating at 33 Mhz (133 MB/sec). This bus replaced EISA as the system bus of PC computers but included a Bridge chip that enabled other types of processors to communicate with it. As a result, PCI is a standard bus on PCs, Macintosh, Sun, and Alpha machines.

PCI is plug and play, using software to configure the boards. It also supports a much larger number of I/O and memory addresses and better direct memory access than its predecessor, EISA. PCI is bus-mastered, meaning that all devices can talk to each other without CPU intervention, thus increasing system performance.

Various permutations of the PCI standard have evolved from increasing the bus width to 64 bit (PCI-64) or upping the effective clock speed to 66 Mhz (PCI-DDR) or 132 Mhz (PCI-QDR). These various flavors of PCI are collectively referred to as PCI-X and were primarily used on workstations and servers.

PCI/PCI-X requires a significant number of wires, or traces, on a motherboard to carry the data. A board needs 42 traces for 32-bit PCI and 82 traces for 64-bit. The bus mastering requires those 42 or 82 traces to be routed across the board and interconnect, creating numerous opportunities for crosstalk or interference. This forces board manufacturers to use boards with more layers to provide additional shielding, increasing costs.

Advanced Graphics Port

AGP is a dedicated video interface that has been the standard on desktop machines for several years. It's a 32-bit system based on the PCI standard, revision 2.1. The initial version, 1x, operated at 66 Mhz for 266 MB/s with direct memory access abilities that PCI doesn’t have. This allows the graphics system to pull additional data from a subset of RAM without the CPU's intervention. The current 8x increases the bandwidth up to 2GB/s and has added more memory access features.

Who needs PCI Express?

Outside of the video bus, current desktops and workstations can quickly become bandwidth-limited, as either a gigabit Ethernet card or RAID controller will use more than 128 MB/s and essentially monopolizes a PCI bus.

PCI Express evolved out of Intel’s 3GIO specification (3rd Generation Input/Output). It's designed to be completely backwards-compatible with existing PCI products, cheaper to manufacture, useful in a variety of roles, and much, much faster.

Achieving the cost reductions, speed increases, and making it useful in different roles required physically redesigning the way connections were made to the bus. PCI Express has its own nomenclature for connections. The interconnects are known as links, with each link consisting of one or more lanes. Each lane has a pair of signal lines, a pair of transfer traces, and a pair of receive traces. The simplest link is X1 and consists of a single lane capable of 250 MB/s. The protocol is expandable to X2 (500 MB/s), X4 (1 GB/s), X8 (2 GB/s), X16 (4 GB/s), and X32 (8 GB/s). Initially, X16 will be the fastest variant released and will be used only for video applications, although some integrated components may utilize X32.

You’ll note that this means an X1 link has only four traces and nearly twice the bandwidth of PCI. The typical board will likely be X4 or X8 and use 16 or 32 traces, a significant reduction in the board manufacturing process.

Additionally, the controllers act as switches, allowing a meshed communication fabric and further reducing the trace length. Existing PCI involves a number of cross connections, meaning those 49 traces run across the board from slot to slot and back to the main bridge. PCI Express simply routes its traces back to the controller, which reduces the signal leakage and the chance of crosstalk. With fewer parallel traces, the motherboards can be made of fewer layers, again reducing expense.

PCI Express controllers will be fully compatible with standard PCI devices. PCI Express slots will have a section that is mechanically and electrically identical to standard PCI, with an extension that handles the additional data transfers, similar to the way EISA added length to the old ISA slots. Since there are a number of applications that do not need the extra bandwidth (modems, 100-Mb Ethernet cards, current sound cards, legacy interface ports), PCI devices will be around for quite some time.

One very useful feature of PCI Express is that the system is hot-swappable. You can remove and install cards without powering down the system. This is a major boom for servers since it allows significant repairs to be performed without taking a production system out of service. With workstation-class PCs acting as servers more and more, especially in clusters, this will be of increasing importance.

Mobile PCI Express

PCI Express has variants intended to eliminate the PC Card CardBus on portable machines. PC cards, or PCMCIA cards as they were originally known, are the standard add-on cards for portable computers. The devices are credit-card sized (85mm x 54mm). Type I cards are 3.3mm thick; Type II cards are 5.0mm thick; and Type III cards are 10.5mm thick. They have a 32-bit interface that provides up to 132 MB/s transfer rates, though typically only Type III cards support the full transfer rate.

Type I PC cards are primarily flash cards or smart cards, and most implement only a 16-bit bus 8 MB/s interface. Early Type II cards used a 16-bit; 20 MB/s interface was often used for modems, network cards, and a few USB 1.0 and RAID controllers. The Type IIIs are almost universally 32-bit and are your WiFi a/G cards or add-on Firewire or USB 2.0 controllers.

Mobile PCI Express comes with the moniker ExpressCard. ExpressCards will initially be seen on laptops, but the eventual goal is to place ExpressCard interfaces on the small form factor desktop PCs that are all the rage. These small systems (similar to the Apple Cube) will have no internal expansion slots but will have several ExpressCard slots for your expansion needs. It's easy to imagine video editing, gigabit Ethernet, or even hard drive ExpressCards making system upgrades truly plug and play.

ExpressCard will come in two flavors—both the same length at 75mm and 5mm thick but either 34mm or 54mm wide. Both ExpressCard/34 and ExpressCard/54 use the same interface, and the ExpressCard/34 can be used in an ExpressCard/54 system. ExpressCards are a bit more complicated in that they are also USB 2.0 devices as well as PCI Express X1. Since all USB and PCI Express devices are hot-swappable, ExpressCards will easily pop in and out of portable machines.

ExpressCard/34 is hoped to be the standard, but the /54 is needed for smartcard/flash memory readers, 1.8" hard drives, and anything that gets too warm to dissipate heat with roughly half the surface area of a PC card. This means we’ll see many first-generation devices released as ExpressCard/54.

Mini-PCI Express

For optional components in laptops or other space-constrained systems, there is Mini-PCI Express. Many laptops and sealed case systems include an internal mini-PCI slot that allows for system customization at the factory without requiring a different motherboard. Integrated WiFi is the most common mini-PCI component today.

Much like mini-PCI, Mini-PCI Express will be for optional components, like WiFi or video cards, in a format not intended to be user-upgradeable. Mini-PCI Express is half the size of mini-PCI, allowing for additional Mini-PCI Express devices or a reduction in system size. While Mini-PCI Express will be X1 in most cases, the specification reserves the capacity to go to X2.

Meet the new bus (same as the old bus)

PCI Express is a massive improvement on PCI and has the advantages of full backwards compatibility. For desktops and servers, there will be a period of transition where video cards are concerned, since AGP cards exist in such large numbers. At least one motherboard manufacturer is planning on releasing a board with both an AGP X8 and a PCI Express X16 slot to mitigate the transition pains.

Laptops will have a similar period of migration away from PC cards to ExpressCards, however, with many laptops having two PC card slots. I wouldn't be surprised to see one PC card and one ExpressCard/54 being the norm for quite some time. With more and more features integrated in portable systems, I'd be surprised by someone having a lack of PC card slots.

With that exception, there should be no issues with the appearance of PCI Express except for increased performance. Oh darn—increased performance. Who needs that?