PCIe - Terminology, Throughput, Root Complex, End Point and, Switch

In my previous post, I introduced the topology and architecture of PCIe. After a careful study, I have gathered a detailed understanding of its terminology and working. Let's have a look!

FIG: Various PCIe slots on a motherboard- PCIe x 4; 16; 1; 16; Conventional 5V 32Bit PCI, respectively. Source: Wikipedia

Terminology:

FIG: PCIe link between two devices consisting of one or more lane. Source: Wikipedia

• Switch, n-point, root complex can be directly connected with the PCIe Link.
• Link contains high-speed serial standard bus, which has a differential signaling. Diffrential signaling are software interrupts sent to a program to indicate an important event like user requests or illegal memory access errors (One transmit pin having two pairs of wires: Transmit +ve and -ve which increases noise immunity). *You can use kill -l on your linux/unix system to view the supported signals.

• For example: If +ve is affected with noise, and in a similar manner if -ve also gets affected with noise. Hence, the reception of the signal, after addition, turns out to be normal.

PCIe Lane Negotiation:

• PCIe link between two devices can be 1 to 32 lanes.
• In a multi-lane link, packet data is stripped across lanes.
• Lane count is automatically negotiated during device initialization.
• It can be restricted by end-point.
• Using this, single lane express device can be used in a multi-lane slot.
• The initialization cycle auto-negotiates the highest mutual lane count during device initialization. e.g.: Bus supports 8 lanes and the device to be connected supports single, then it auto-negotiates the support and establishes a communication.
• Links can also be connected down dynamically and use fewer lanes as well.

PCIe Throughput:

Version	x1 (Pin Count 18)	x2	x4 (32)	x8 (49)	x16 (82)
PCIe 1.x	0.25	0.5	1.0	2.0	4.0
PCIe 2.x	0.5	1.0	2.0	4.0	8.0
PCIe 3.x	1.0	2.0	4.0	8.0	16.0
PCIe 4.x	2.0	4.0	8.0	16.0	32.0

• Above table: 0.25 = 250mbps; 0.5 = 500mbps; 1.0 = 1gbps and so on.

Root Complex:

• PCIe Root Complex is the Root of a hierarchy that connects with the CPU and Memory sub-systems.
• Other than the Root Complex, such as an end-point or a switch do not have the connection with CPU or Memory. All connections pass through Root Complex sub-system.
• It supports one or more PCIe ports.
• We have multiple interfaces in Root Complex. Each interface defines a separate hierarchy domain.
• To transfer larger packets to the end-point from the memory, Root Complex splits a packet into smaller size packets.
• generates configuration/IO Requests as a requestor. Requestor: creates request; Responder: generates a response.
• Root Complex also generates Locked Requests as the requestor.

PCIe End Point:

• It must be a function of a type 00h.
• Must support configuration requests as a completor.
• It does not depend on OS allocation of I/O resources.
• In this we have Message Signaled Interrupt (MSI), to interrupt the requests.
• If prefetch bit is set, 64-Bit addressing is supported.
• If prefetch bit is not set, 32-Bit addressing is supported.
• BAR minimum memory address range is 128 bytes (Base Address Register).
• Should appear in one of the domains. 

PCIe Switch:

FIG: PCIe card containing PCIe Switch covered by a Heat Sink - Creates multiple endpoints out of one endpoint with multiple device sharing. Source: Wikipedia

• Has two ports called upstream and downstream port. 

FIG: A PCIe Switch.

• If we go upwards through an upstream port it reaches to the Root Complex. If we go downwards we will, somehow, reach to the endpoints (If there is an endpoint on the bus).
• The downstream port cannot be connected to the Root Complex. Similarly, the upstream port cannot be connected with the endpoint.
• The switch appears to be two or more logical PCI-to-PCI bridges.
• The switch has to do the address based routing except when in Multicast. e.g.: if the Root Complex accesses the device A, the packet should get routed to that endpoint.
• It forwards all type of transaction layer packets between a set of ports.
• In case we have a switch with four ports, the arbitration will happen over the four ports in round robin or weighted round robin manner, depending upon the scenario.
• It does not split the packet into smaller packets.
• switch port supports flow control specification (for e.g. maintaining the requests).