PCIe - Layering

FIG: Open-end PCIe x1 for longer PCIe cards supported @ x1 Speeds. Source: Wikipedia

Previously, I discussed the details of a PCIe switch and how it is used for address-based routing and how it is used to control the flow by maintaining the requests. To further this concept now I am going to discuss the PCIe to PCI/PCI-X Bridge.

PCIe to PCI/PCI-X Bridge:

• It provides a connection between PCIe and PCI/PCI-X.
• If our system needs to be connected with the PCI devices, then we need this bridge which converts the high-speed serial bus of PCIe to the address and data multiplexed bus by using a bridge. 

PCI Compatible Model:

• It has the ability to calculate one by one and configure PCIe similar to PCI.
• Boots the existing OS without modifications.
• Existing I/O  device drivers can be used without any modifications.
• Configure/enable PCIe functionality by PCI configurations.

PCIe Layering:

FIG: All PCI express Layers.

• PCIe is a packet-based communication between the root complex and Endpoints.
• Since it is traversing via serial packet-based communication, it is structured into Layers.
• Data is sent bit-by-bit to the other end. Packetizing and De-Packetizing are done in Layers.
• Data is sent from Endpoints to root complex.
• PCIe has THREE Layers as shown in the above figure.
• When the transaction is initiated by the CPU, data is converted to the Transaction Layer packet. The transaction Layer packet will act as data for the Data-Link Layer packet, which further acts as data for Physical Layer packet. This happens in the opposite manner at the Reception end.
• At Rx, data is given to the Root Complex from the Transaction Layer after receiving the packet for execution purpose.
• PCIe Layering uses packets to communicate data between components. For e.g. data communication from Root Complex to a Switch and from Switch to Endpoint and vice versa.
• Packets are formed in Transaction, Data Link Layer at Tx and, at the receiving side, the reverse process occurs and the packets get transformed.
• The transmitted packet flow through other Layers, they get extended.

PCIe Packet Flow:

FIG: PCIe Packet Flow is shown above.

• 'Data' and 'ECRC' i.e. EndPoint Cycle Redundancy Check or EndPoint CRC are both optional PCIe features.
• LCRC: Link CRC.
• Physical Layer adds up Framing feature to send serial bit-by-bit data to the other end.
• Hence, a complete Transaction Layer packet is data for Data Link Layer packet and Data Link Layer packet is data for Physical Layer packet.

Transaction Layer:

• The Upper Layer of PCIe architecture: assembles and disassembles Transaction Layer Packets (TLP).
• TLPs are used to communicate transactions, such as Read & Write, I/O R & W, etc.
• Responsible for managing a credit-based flow control mechanism between 2 devices (the mechanism is based on flow control for each leg).
• Each packet has a unique identifier that enables the response to a correct originator.
• Supports four address space:
• Memory, I/O, Configuration, Message (message space supports prior side-band signals, interrupts, power management requests and, system-generated events).

Transaction Layer Services:

• Responsible for generating and receiving TLPs.
• Exchanges flow control information between two sides of the link irrespective of hardware and software initiated power management.
• For e.g. initiating the power management when the system goes to 'sleep' mode by removing the VGA cord.
• Stores the Link configuration info generated by CPU and stores Link capabilities generated by Physical Layer.
• Packet generation and Process services.
• Manages Flow Control and Power Management services.

Data Link Layer:

• The Middle Layer of PCIe architecture / Slack: Intermediate between Transaction and Physical Layer.
• Link management and data integrity, error detection & correction.
• Accepts TLPs, calculates and applies data protection code, TLP Sequence number.
• Checks the integrity of received TLPs.
• On TLP error, requests retransmission of TLPs until correctly received or Link is determined to have failed.
• Generates & consumes packets used for Link management functions.
• These packets are referred to as Data Link Layer Packets (DLLP).

Data Link Layer Services:

• Responsible for reliable info exchange with the opposite link.
• Checks sequence number and data integrity, so that there is no data loss between the links.
• Initialization and Power management services.
• Active/Reset/Disconnected/Power Managed states.
• Accepts power state requests and conveys it to the Physical Layer.
• Data Protection, Error Checking and Retry services.

Physical Layer:

• Includes all the hardware circuitry for interface operation (Parallel to Serial convertor, Differential signaling circuits, Buffered management, etc).
• Logical functions such as interface initialization and maintenance are handled in Physical Layer.
• Converts information from DLL into a serialized format.
• Serialized format into information for DLL.
• Supports future performance enhancements via speed upgrades.

Physical Layer Services:

• Interface initialization, maintenance control, and status tracking.
• Symbol and special ordered set generation
• Symbols transmission and alignment
• Transmission circuits
• Reception circuits
• Elastic Buffer at receiving side

Inter-Layer Interface:

• Transaction/Data Link Interface:
• Byte or Multi-Byte data sent across the link.
• TLP framing information for the received byte.
• Requested power state for the link.
• Link Status Information.
• Data Link/Physical Interface:
• Byte or Multi-Byte wide data received from PCIe Link.
• TLP and DLLP framing information for data.
• Indication of errors detected by the physical layer.
• Connection status information update.

Transaction types, Address space:

• 4 types of Address Spaces.

Address Space	Transaction type	Usage
Memory	Read/Write	Transfer data to/from a memory-mapped location
I/O	Read/Write	Transfer data to/from an IO mapped location
Configuration	Read/Write	Device function configuration setup
Message	Baseline	From event signaling mechanism to general-purpose messaging, interrupt messaging, power management

Memory Transactions:

• Memory transaction are given below
• Read request/completion. Every Read request will have a response of completion with data.
• Write request.
• Atomic operation request/completion.
• Two different address formats
• Short address format: 32-bit address
• Long address format: 64-bit address

I/O Transactions:

• I/O Read and Write can access I/O space on CPU. It supports I/O space compatible with the legacy devices.
• In the future, I/O transactions might disapprove or abhor the use of I/O space in the upcoming PCIe versions. Reason: Usage of I/O transaction has reduced recently, for e.g. the use of Serial and Parallel ports in earlier CPUs has now become obsolete. 
• The following are I/O transaction types:
• Read request/completion
• Write request/completion
• Address format support:
• Short: 32-bit address

Configuration Transaction:

• It is used to access the configuration registers of function in devices. 
• The config space has structured information (Vendor ID; Device ID; etc).
• Config transaction types:
• Read request/completion
• Write request/completion

Message Transactions:

• Supports in-band communication of events between devices.
• Vendor defined messages use specific message codes and are more specific to the platform (means, we need to define the particular device by the Vendor that how the message format is being received at the Root Complex).