PCI Express
|
Overview
The PCIe physical layer consists of a network of serial interconnects. A hub on the mainboard acts as a crossbar switch allowing point-to-point device interconnections to be rerouted on the fly. This dynamic point-to-point connection behavior leads to parallelism since more than one pair of devices may communicate with each other at the same time. (In contrast, older PC interfaces had all devices permanently wired to the same bus; therefore, only one device could talk at a time.) This is similar to the difference between conversing over a telephone where you can only call one person at a time, and conversing in a meeting, where you can talk to a person beside you directly. The format also allows channel grouping, where multiple lanes are bonded to a single device pair in order to provide higher bandwidth.
The bonded serial format was chosen over a traditional parallel format due to the phenomenon of timing skew. Timing skew is a direct result of the limitations imposed by the speed of an electrical signal traveling down a wire, which it does at a finite speed. Because different traces in an interface have different lengths, parallel signals transmitted simultaneously from a source arrive at their destinations at different times. When the interconnection clock rate rises to the point where the wavelength of a single bit is less than this difference in path length, the bits of a single word do not arrive at their destination simultaneously, making parallel recovery of the word difficult. Thus, the speed of the electrical signal, combined with the difference in length between the longest and shortest trace in a parallel interconnect, leads to a naturally imposed maximum bandwidth. Serial channel bonding avoids this issue by not requiring the bits to arrive simultaneously. PCIe is just one example of a general trend away from parallel buses to serial interconnects. For other examples, see HyperTransport, Serial ATA, USB, SAS, FireWire or RapidIO. The multichannel serial design also increases flexibility by allowing slow devices to be allocated fewer lanes than fast devices.
PCIe is supported primarily by Intel, which started working on the standard as the Arapahoe project after pulling out of the InfiniBand system. PCIe is intended to be used as a local interconnect only. It was designed to be software compatible with the preexisting PCI standard, making the conversion of PCI cards and systems to PCI Express as simple as replacing the physical layer without requiring a change to the supporting software. The increased bandwidth on PCI Express has led to unification, as it is fast enough to replace almost all existing internal buses, including AGP and PCI. Intel envisions a single PCI Express controller talking to all external devices in the future, as opposed to the northbridge/southbridge solution used in current machines.
Unlike preceding PC expansion interface standards, PCIe is a point-to-point "bus". This type of connection removes the need for "arbitrating" the bus or waiting for the bus to free. This means that while standard PCI-X (133 MHz 64 bit) and PCIe x4 have roughly the same data transfer rate, PCIe x4 will give better performance if multiple device pairs are communicating simultaneously or if communication within a single device pair is bidirectional.
Physical Layer
The PCIe Physical Layer (PHY) (PCIEPHY , PCI Express PHY or PCIe PHY) specification is divided into two sublayers, corresponding to electrical and logical specifications. The logical sublayer is sometimes further divided into a MAC (Media Access Control) sublayer and a PCS (Physical Coding Sublayer), although this division is not formally part of the PCIe specification. A specification published by Intel, the PHY Interface for PCI Express (PIPE)[2] , defines the MAC/PCS functional partitioning and the interface between these two sublayers. The PIPE specification also identifies the PMA (Physical Media Attachment) layer, which includes the Serializer/Deserializer and other analog circuitry; however, since SerDes implementations vary greatly among ASIC vendors, PIPE does not specify an interface between the PCS and PMA.
At the electrical level, each lane consists of two unidirectional LVDS or PCML pairs at 2.52½ Gbit/s. Transmit and receive are separate differential pairs, for a total of 4 data wires per lane.
A connection between any two PCIe devices is known as a "link", and is built up from a collection of 1 or more lanes. All devices must minimally support single-lane (x1) link. Devices may optionally support wider links composed of 2, 4, 8, 12, 16, or 32 lanes. This allows for very good compatibility in two ways:
- a PCIe card will physically fit (and work correctly) in any slot that is at least as large as it is (e.g. an x1 sized card will work in any sized slot);
- a slot of a large physical size (e.g. x16) can be wired electrically with fewer lanes (e.g. x1, x4, or x8) as long as it provides the power and ground connections required by the larger physical slot size.
In both cases, PCIe will negotiate the highest mutually supported number of lanes.
It is often not possible to place a physically larger PCIe card (e.g. a 16x sized card) into a smaller slot, even though the two would be signal-compatible if it were possible. Some motherboards have open-ended PCIe slots which allow for a physically larger card to be inserted in a smaller PCIe slot.
The width of a PCIe connector is 8.8 mm, while the height is 11.25 mm, and the length is variable. The 'minor' half of the connector is 11.65 mm in length and contains 22 pins, while the length of the 'major' half is variable. The thickness of the card going into the connector is 1.8mm.
| Lanes | Pins Total | Pins in 'major' half | Total Length | Length of 'major' half |
|---|---|---|---|---|
| x1 | 36 | 14 | 25 mm | 7.65 mm |
| x4 | 64 | 42 | 39 mm | 21.65 mm |
| x8 | 98 | 76 | 56 mm | 38.65 mm |
| x16 | 164 | 142 | 89 mm | 71.65 mm |
Data transmission
PCIe sends all control messages, including interrupts, over the same links used for data. The serial protocol can never be blocked, so latency is still comparable to PCI, which has dedicated interrupt lines.
Data transmitted on multiple-lane links is interleaved, meaning that each successive byte is sent down successive lanes. The PCIe specification refers to this interleaving as "data striping." While requiring significant hardware complexity to synchronize (or deskew) the incoming striped data, striping can significantly increase the throughput of the link. Due to padding requirements, striping may not necessarily reduce the latency of small data packets on a link.
As with all high data rate serial transmission protocols, clocking information must be embedded in the signal. At the physical level, PCI Express utilizes the very common 8b/10b encoding scheme to ensure that strings of consecutive ones or consecutive zeros are limited in length. This is necessary to prevent the receiver from losing track of where the bit edges are. In this coding scheme every 8 (uncoded) payload bits of data are replaced with 10 (encoded) bits of transmit data, consuming an extra 25% of the overall electrical bandwidth.
Many other protocols (such as SONET) use a different form of encoding known as "scrambling" to embed clock information into data streams. The PCI Express specification also defines a scrambling algorithm, but it is used to reduce EMI (Electromagnetic interference) by preventing repeating data patterns in the transmitted data stream.
Signaling rate
The first-generation PCIe transfers data at a 2.5 GT/s (gigatransfer per second) signaling rate per lane. PCIe version 2.0 provides an increase in the signaling rate to 5 GT/s per lane. A third-generation PCIe specification is in development with the goal of further increasing the rate.
Data Link Layer
The Data Link Layer implements the sequencing of the Transaction Layer Packets (TLPs) that are generated by the Transaction Layer, data protection via a 32-bit cyclic redundancy check code (CRC, known in this context as LCRC) and an acknowledgment protocol (ACK and NAK signaling). TLPs that pass an LCRC check and a sequence number check result in an acknowledgment, or ACK, while those that fail these checks result in a negative acknowledgment, or NAK. TLPs that result in a NAK, or timeouts that occur while waiting for an ACK, result in the TLPs being replayed from a special buffer in the transmit data path of the Data Link Layer. This guarantees delivery of TLPs in spite of electrical noise, barring any malfunction of the device or transmission medium.
ACK and NAK signals are communicated via a low-level packet known as a data link layer packet, or DLLP. DLLPs are also used to communicate flow control information between the transaction layers of two connected devices, as well as some power management functions.
Transaction Layer
PCI Express implements split transactions (transactions with request and response separated by time), allowing the link to carry other traffic while the target device gathers data for the response.
PCI Express utilizes credit-based flow control. In this scheme, a device advertises an initial amount of credit for each of the receive buffers in its Transaction Layer. The device at the opposite end of the link, when sending transactions to this device, will count the number of credits consumed by each TLP from its account. The sending device may only transmit a TLP when doing so does not result in its consumed credit count exceeding its credit limit. When the receiving device finishes processing the TLP from its buffer, it signals a return of credits to the sending device, which then increases the credit limit by the restored amount. The credit counters are modular counters, and the comparison of consumed credits to credit limit requires modular arithmetic. The advantage of this scheme (compared to other methods such as wait states or handshake-based transfer protocols) is that the latency of credit return does not affect performance, provided that the credit limit is not encountered. This assumption is generally met if each device is designed with adequate buffer sizes.
First-generation PCIe is often quoted to support a data rate of 250 MB/s in each direction, per (x1) lane. This figure is a calculation from the physical signaling rate (2.5 Gbaud) divided by the encoding overhead (10 bits per byte.) This means a sixteen lane (x16) PCIe card would then be theoretically capable of 250 MB/s * 16 = 4 GB/s in each direction. While this is correct in terms of data bytes, more meaningful calculations will be based on the usable data payload rate, which depends on the profile of the traffic, which is a function of the high-level (software) application and intermediate protocol levels. Like other high data rate serial interconnect systems, PCIe has a protocol and processing overhead due to the additional transfer robustness (CRC and Acknowledgments). Long continuous unidirectional transfers (such as those typical in high-performance storage controllers) can approach >95% of PCIe's raw (lane) data rate. These transfers also benefit the most from increased number of lanes (x2, x4, etc.) But in more typical applications (such as a USB or Ethernet controller), the traffic profile is characterized as short data packets with frequent enforced acknowledgments. This type of traffic reduces the efficiency of the link, due to overhead from packet parsing and forced interrupts (either in the device's host interface or the PC's CPU.) This loss of efficiency is not particular to PCIe.
No comments:
Post a Comment