About TLP payload size.

Comments and questions to the author of the "Down to the TLP" posts (Eli Billauer)

About TLP payload size.

Postby Guest » Wed May 02, 2018 12:35 am

Great article! I got a couple of questions about TLP payload size which commonly uses 128B, 256B and 512B.
1. Does MRd and Mwr have to use the same payload size on the same machine or they can be different?
2. Does, generally, larger(256B vs. 128B) TLP payload size work better from performance perspective by reducing the number of TLP transactions? Or the TLP payload size also has dependency on the device capability itself? I know some BIOS may allow customizing TLP payload size but what I find out is they typically prefer using smaller 128B as the default payload size. Not sure if this is for safety reasons because some devices may not support larger TLP payload size?

Thanks!
Guest
 

Re: About TLP payload size.

Postby support » Wed May 02, 2018 6:02 am

Hello,

The said figures relate to the maximal payload size. Any given TLP (of the type carrying payload) can have any length from zero to the maximum.

The advantage of a large payload is that the proportion of the overhead data (TLP and link layer headers) gets smaller, so it's possible to utilize the raw bandwidth better. But the PCIe link is rarely the bottleneck in common PC cards.

During the first few years PCIe was available, motherboards only supported 128 bytes of Maximal Payload Size. As a result, the vast majority of hardware made its initial mileage in this mode. It might be, that some hardware misbehaved when the limit was lifted towards 256 or 512 bytes. Given the relatively small advantage of larger payload vs. maybe some buggy hardware that doesn't handle longer packets well, I can speculate the reason why some BIOSes set the limit to 128 by default.

Either way, the Maximal Payload Size is negotiated between the host and peripheral, so if a device supports only 128 bytes, that's the end of it, regardless of the host's capabilities (and vice versa).

Regards,
Eli
support
 
Posts: 570
Joined: Tue Apr 24, 2012 3:46 pm

Re: About TLP payload size.

Postby Guest » Wed May 02, 2018 4:53 pm

Thanks for your reply. One of particular interesting thing I found on a HP work station is, upon capturing PCIe traces to the same PCIe NVMe SSD, MRd TLP packet is using 128B payload size while MWr TLP packet is using 256B payload size. Does the negotiation between host and the end device about maximum TLP payload size differentiate between MRd and MWr? I never saw any article or even PCIe spec mention TLP maximum payload size may vary between MRd and MWr. Any thoughts here?
Guest
 

Re: About TLP payload size.

Postby support » Wed May 02, 2018 7:18 pm

Hello,

The Maximal Payload Size relates to all TLPs. In your case it was set to 256 or higher. The fact that packets in one direction had shorter payloads than the other is just some kind of implementation choice. It says nowhere that the maximal payload has to be utilized for any or all packets.

Regards,
Eli
support
 
Posts: 570
Joined: Tue Apr 24, 2012 3:46 pm

Re: About TLP payload size.

Postby Guest » Wed May 02, 2018 11:50 pm

Btw, If the root port's max payload size is set to 128B, does that mean the effective max payload size for every peripheral device('s uplink) connected to that root port will be subject to that root port limit regardless their own max payload size settings? Thanks.
Guest
 

Re: About TLP payload size.

Postby support » Thu May 03, 2018 2:59 am

Hello,

Strictly speaking, there is no such thing as the root port's max payload size. There is, of course a limitation on what the hardware of the root port can support, but it's up to the software (often the BIOS) to decide what max payload to allocate to each device, based upon the limit given in its relevant configuration register (the Device Control Register), and informing it of the verdict by writing to the same.

So in principle, the host's software may have a blacklist of devices, for which it avoids large Maximal Payload Size assignments. Even though I doubt such exists.

But yes, a more sane situation is that the host's hardware has a Max Payload Size of say 128 bytes, and then all devices will have the same 128 byte limit, regardless of the higher limit they may support.

Regards,
Eli
support
 
Posts: 570
Joined: Tue Apr 24, 2012 3:46 pm

Re: About TLP payload size.

Postby Guest » Thu May 03, 2018 10:05 pm

One more question here, do you know if MRd TLP payload size has to eventually align with Completion with CplD Read Completion Boundary(RCB) size. That means, if root complex uses 128Byte RCB, even though MPS is 256Byte, Mrd CplD TLP still uses 128Byte. If that is true, that may explain why I saw different MRd(128Bytes) and MWr(256Bytes) TLP payload size on the same machine based on captured PCIe traces. Thanks!
Guest
 

Re: About TLP payload size.

Postby support » Fri May 04, 2018 6:49 am

Hello,

There is no need to explain why the packet sizes were different. It's perfectly up to the designer to decide that completions packets won't carry more than 128 bytes, regardless of the max payload size for whatever reason. Or the size of the write packets. The behavior you've observed is within spec.

As for RCB, it's briefly discussed on this page:

http://xillybus.com/tutorials/pci-expre ... al-guide-2

However this is irrelevant, as it relates to boundaries. A 256-byte payload starting at a 128-byte boundary ends at a 128-byte boundary, so there's nothing stopping the use of 256-byte payload completion packets with a 128-byte (or 64-byte) RCB.

Regards,
Eli
support
 
Posts: 570
Joined: Tue Apr 24, 2012 3:46 pm

Re: About TLP payload size.

Postby Guest » Mon May 07, 2018 6:55 pm

In PCIe Header, under PCIe Device Capabilities register, there is Max_Payload_Size Supported(Read Only). Based on the spec, it says "This field indeicates the maximum payload size that functionality may support for TLPs"; Under PCIe Device Control register, there is also Max_Payload_Size(Read/Write). It says "this field sets the maximum TLP payload size for the function. As a receiver, the Function shall handle TLPs as large as the set vcalue. As a transmitter, the Function shall not generate TLPs exceeding the set value. Functions that only support 128 byte max payload size are permitted to hardware this field 0h". Do you know what the difference between these two MPs besides one is read-only and the other is read/write? Thanks.
Guest
 

Re: About TLP payload size.

Postby support » Tue May 08, 2018 3:42 am

Hello,

What you've cited is the registers of the negotiation mechanism: The read-only register is where the device announces its capability (maximal payload size it's supporting) and the read/write register is where the host writes the max payload size in effect (i.e. the host's decision).

Regards,
Eli
support
 
Posts: 570
Joined: Tue Apr 24, 2012 3:46 pm


Return to General PCIe

cron