Data Link Layer

Transmitter
Packet-Based Transaction
Why Use A Packet-Based Transaction Protocol :

● Some early bus protocols (e.g. PCI) allow transfers of indeterminate (and unlimited)
size, making identification of payload boundaries impossible until the end of the
transfer. It is difficult for the sender of data to calculate and send a checksum or CRC
covering an entire payload.
● In contrast, each PCI Express packet has a known size and format, and the packet
header--positioned at the beginning of each DLLP and TLP packet. The size of any
data payload is conveyed in the TLP header Length field.
● Each TLP and DLLP packet sent is framed with a Start and End control symbol, clearly
defining the packet boundaries to the receiver by using Framing Symbols.
● CRC value “protects” the entire packet, In addition to CRC, TLP packets also have a
packet sequence number appended to them by the transmitter so that if an error is
detected at the receiver, the specific packet(s) which were received in error may be
resent.
DATA LINK LAYER PACKETS

Types Of DLLPs :

1. TLP Acknowledgement Ack/Nak DLLPs

2. Flow Control Packet DLLPs

3. A 1 DW core (4 bytes) consisting of the one byte

Type field and three additional bytes of attributes.
The attributes vary with the DLLP type.
4. A 16 bit CRC value which is calculated based on the
DW core contents, then appended to it.
DLLP Format
Ack/Nak DLLPs
ACK/NAK PROTOCOL DETAILS

Transmitter Protocol Details :

● Sequence Number :
Before a transmitter sends TLPs delivered by the Transaction Layer, the Data Link Layer appends a 12-bit
Sequence Numbers to each TLP. The Sequence Number is generated by the 12-bit NEXT_TRANSMIT_SEQ counter.
The counter is initialized to 0 at reset, or when the Data Link Layer is in the inactive state. It increments after each
new TLP is transmitted until it reaches its maximum value of 4095, and then rolls over to 0. For each new TLP sent,
the transmitter appends the Sequence Number from the NEXT_TRANSMIT_SEQ counter.

● 32-Bit LCRC :
The transmitter also appends a 32-bit LCRC (Link CRC) calculated based on TLP contents which include the
Header, Data Payload, ECRC and Sequence Number.

● Replay (Retry) Buffer :

Before a device transmits a TLP, it stores a copy of the TLP in a buffer associated with the Data Link Layer
referred to as the Replay Buffer (the specification uses the term Retry Buffer). Each buffer entry stores a complete
TLP with all of its fields including the Header (up to 16 bytes), an optional Data Payload (up to 4KB), an optional
ECRC (up to four bytes), the Sequence Number (12-bits wide, but occupies two bytes) and the LCRC field (four
bytes).
The transmitter's response to reception of an ACK DLLP include:

● Load ACKD_SEQ register with AckNak_Seq_Num[11:0] of the ACK DLLP.

● Reset the REPLAY_NUM counter and REPLAY_TIMER to 0.
● Purge the Replay Buffer as described below.

The transmitter's response to reception of a NAK DLLP include:

● Reset REPLAY_NUM and REPLAY_TIMER to 0 only if the NAK DLLP's AckNak_Seq_Num[11:0] is later than the
current ACKD_SEQ value (forward progress is made in transmitting TLPs).
● Load ACKD_SEQ register with AckNak_Seq_Num[11:0] of the NAK DLLP.

● TLP Replay :

The transmitter blocks the delivery of new TLPs by the Transaction Layer. It then replays (re-sends or retries)
the contents of the Replay Buffer starting with the earliest TLP first.ACK DLLPs or NAK DLLPs received during
replay must be processed. This means that the transmitter must process the DLLPs and, at the very least, store them
until the replay is finished. After replay is complete, the transmitter evaluates the ACK or NAK DLLPs and performs
the appropriate processing.
 ● Repeated Replay of TLPs :
The transmitter uses a 2-bit Replay Number counter, referred to as the REPLAY_NUM counter, to keep track
of the number of replay events. Reception of a NAK DLLP increments REPLAY_NUM. This counter is initialized to 0
at reset, or when the Data Link Layer is inactive. It is also reset if an ACK or NAK DLLP is received with a later
Sequence Number than that contained in the ACKD_SEQ register.

What Happens After the Replay Number Rollover ?

A transmitter's Data Link Layer triggers the Physical Layer to re-train the Link. The
Physical Layer Link Training and Status State Machine (LTSSM) enters the Recovery State. The
Replay Number Rollover error bit is set.

Receiver Protocol Details :

● TLP Received at Physical Layer :

TLPs received at the Physical Layer are checked for STP and END framing errors as well as other receiver
errors such as disparity errors. If there are no errors, the TLPs are passed to the Data Link Layer. If there are any
errors, the TLP is discarded and the allocated storage is freed up. The Data Link Layer is informed of this error so
that it can schedule a NAK DLLP.
● Received TLP Error Check :
The receiver accepts TLPs from the Link into a receiver buffer and checks for CRC errors. The receiver
calculates an expected LCRC value based on the received TLP (excluding the LCRC field) and compares this value
with the TLP's 32-bit LCRC. If the two match, the TLP is good. If the two LCRC values do not match, the received TLP
is bad and the receiver schedules a NAK DLLP to be returned to the remote transmitter.

● Next Received TLP's Sequence Number :

The receiver keeps track of the next expected TLP's Sequence Number via a 12-bit counter referred to as the
NEXT_RCV_SEQ counter. This counter is initialized to 0 at reset, or when the Data Link Layer is inactive. This
counter is incremented once for each good TLP that is received and forwarded to the Transaction Layer. The counter
rolls over to 0 after reaching a value of 4095.

The receiver uses the NEXT_RCV_SEQ counter to identify the Sequence Number that should be in the next
received TLP. If a received TLP has no LCRC error, the device compares its Sequence Number with the
NEXT_RCV_SEQ count. If these two numbers matches, the receiver accepts the TLP, forwards the TLP to the
Transaction Layer, increments the NEXT_RCV_SEQ counter and is ready for the next TLP.

At this point, the receiver can schedule an ACK DLLP with the Sequence Number of the received TLP. When
the receiver schedules an ACK DLLP to be returned to the remote transmitter, The receiver continues to receive TLPs
and as long as there are no detected errors
 ● NAK Scheduled Flag :

The receiver implements a Flag bit referred to as the NAK_SCHEDULED flag. When a receiver detects a TLP
CRC error, or any other non-CRC related error that requires it to schedule a NAK DLLP to be returned, the receiver
sets the NAK_SCHEDULED flag and clears it when the receiver detects replayed TLPs from the transmitter for which
there are no CRC errors.

A received :TLP's Sequence Number may not match the NEXT_RCV_SEQ count.the
NEXT_RCV_SEQ counter is not incremented.
● TLP Sequence Number is logically less than NEXT_RCV_SEQ count (earlier Sequence Number). This
situation results when a duplicate TLP is received as the result of a replay event. The duplicate TLP is
discarded. NEXT_RCV_SEQ count is not incremented. An ACK DLLP is scheduled so that the
transmitter can purge its Replay Buffer of the duplicate TLP(s).

● TLP Sequence Number is logically greater than NEXT_RCV_SEQ count (later Sequence Number). This
situation results when one or more TLPs are lost en route. The receiver schedules a NAK DLLP for
return to the transmitter if NAK_SCHEDULED flag is clear
Flow Control DLLPs
Why Flow Control Mechanism ?

● Before a transaction packet can be sent across a link to the receiving

port, the transmitting port must verify that the receiving port has
sufficient buffer space to accept the transaction to be sent. In many
other architectures including PCI and PCI-X, transactions are
delivered to a target device without knowing if it can accept the
transaction. If the transaction is rejected due to insufficient buffer
space, the transaction is resent (retried) until the transaction
completes. This procedure can severely reduce the efficiency of a bus.
● The link Flow Control mechanism uses a credit-based mechanism that
allows the transmitting port to check buffer space availability at the
receiving port. During initialization each receiver reports the size of its
receive buffers (in Flow Control credits) to the port at the opposite end
of the link. The receiving port continues to update the transmitting
port regularly by transmitting the number of credits that have been
freed up. This is accomplished via Flow Control DLLPs.
FLOW CONTROL BUFFERS
Device A
Each VC Flow Control buffer at the receiver is managed for each
category of transaction flowing through the virtual channel. These
categories are:

● Posted Transactions — Memory Writes and Messages

● Non-Posted Transactions — Memory Reads, Configuration Reads and

Writes, and I/O Reads and Writes
● Completions — Read Completions and Write Completions

In addition, each of these categories is separated into header and data

portions of each transaction. Flow control operates independently for
each of the six buffers listed below

● Posted Header
● Posted Data
● Non-Posted Header
● Non-Posted Data
● Completion Header
● Completion Data
Flow Control Credits :
● Header FCCs — maximum header size + digest
○ 4 DWs for completions
○ 5 DWs for requests
● Data FCCs — 4 DWs (aligned 16 bytes)

Maximum Flow Control Buffer Size :

128 Credits for headers :
● 2,560 bytes Request Headers @ 20 bytes/credit
● 2048 bytes for completion headers @ 16 bytes/credit

2048 Credits for data :

● 32KB @ 16 bytes/credit
The Flow Control Elements :

Credit Allocated Counter :

This counter tracks the total Flow Control credits that have been allocated. As the buffer fills, the amount of
available buffer space decreases until transactions are removed from the buffer. The number of Flow Control credits
associated with each transaction removed from the buffer is added to the CREDIT_ALLOCATED counter.

Credit Limit Register :

This register is initialized by the receiving device when it sends Flow Control initialization packets to report the
size of the corresponding Flow Control receive buffer. Following initialization, Flow Control update packets are sent
periodically to add more Flow Control credits as they become available at the receiver.

Credit Consumed Counter :

Tracks the size of all transactions sent from the VC buffer in Flow Control credits.

Credits Received Counter (optional):

This counter keeps track of the total size of all data received from the transmitting device and placed into the
Flow Control buffer. When flow control is functioning properly, the CREDITS_RECEIVED count should be the same
as CREDITS_CONSUMED count at the transmitter and be equal to or less than the CREDIT_ALLOCATED count. If
this is not true, then a flow control buffer overflow has occurred and error is detected.
RECOMMENDED PRIORITY TO SCHEDULE PACKETS :

A device may have many types of TLPs, DLLPs and PLPs to transmit on a given Link. The following
is a recommended but not required set of priorities for scheduling packets:

1. Completion of any TLP or DLLP currently in progress (highest priority).

2. NAK DLLP.
3. ACK DLLP.
4. FC (Flow Control) DLLP.
5. Replay Buffer re-transmissions.
6. TLPs that are waiting in the Transaction Layer.
7. All other DLLP transmissions (lowest priority)
 Device A Device B
TL TL TL TL
Tx Rx Tx Rx

DLL DLL DLL DLL

Tx Rx Tx Rx

PL PL PL PL
Tx Rx Tx Rx
Data Link Layer Errors

Link layer error checks occur within a device involved in delivering the transaction between the requester
and completer functions. This includes the requesting device, intermediate switches, and the completing
device. Checks performed at the link layer include:

● LCRC check failure for TLPs

● Sequence Number check for TLP s
● LCRC check failure for DLLPs
● Replay Time-out
● Replay Number Rollover
● Data Link Layer Protocol errors
● Problem: One or more TLPs are lost en route to the receiver.
Solution: The receiver performs a sequence number check on all received TLPs. The receiver expects TLPs to arrive
with each TLP that has an incremented 12-bit Sequence Number from that in the previous TLP. If one or more TLPs
are lost en route, a TLP will have a Sequence Number issued later than expected Sequence Number reflected in the
NEXT_RCV_SEQ count. The receiver schedules a NAK DLLP with a Sequence Number = NEXT_RCV_SEQ count -
1. Transmitter replays the Replay Buffer contents.
● Problem: Receiver returns an ACK DLLP, but it is corrupted en route to the transmitter. The remote Transmitter
detects a CRC error in the DLLP. In fact, the transmitter does not know that the malformed DLLP just received is
supposed to be an ACK DLLP. All it knows is that the packet is a DLLP.
Solution:
- Case 1: The Transmitter discards the DLLP. A subsequent ACK DLLP received with a later Sequence
Number causes the transmitter Replay Buffer to purge all TLPs with equal and earlier generated Sequence Numbers.
The transmitter never knew that anything went wrong.

- Case 2: The Transmitter discards the DLLP. A subsequent NAK DLLP received with a later generated
Sequence Number causes the transmitter Replay Buffer to purge TLPs with equal to an earlier Sequence Numbers.
The transmitter then replay all TLPs with later Sequence Numbers till the last TLP in the Replay Buffer. The
transmitter never knew that anything went wrong.
● Problem: ACK or NAK DLLP for received TLPs are not returned by the receiver by the proper
ACKNAK_LATENCY_TIMER time-out. The associated TLPs remain in the transmitter Replay Buffer.
Solution: The REPLAY_TIMER times-out and the transmitter replays its Replay Buffer.
CRC Calculation Example :
THANK YOU