PCIe Max Payload Size

Comments and questions to the author of the "Down to the TLP" posts (Eli Billauer)

PCIe Max Payload Size

Postby Guest » Wed May 08, 2013 7:26 pm

I'm wondering if there is a tool or utility you can run to verify if the max payload size on the root port is the same as the max payload size on all intermediate bridges and devices. Can lspci do this?
Guest
 

Re: PCIe Max Payload Size

Postby support » Wed May 08, 2013 8:05 pm

Yes, lspci does indeed supply the information. For example,

    # lspci -vv
    (...)
    01:00.0 Class ff00: Xilinx Corporation Xillybus Generic FPGA core
    Subsystem: Xilinx Corporation Xillybus Generic FPGA core
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 0, Cache Line Size: 4 bytes
    Interrupt: pin ? routed to IRQ 41
    Region 0: Memory at fdaff000 (64-bit, non-prefetchable) [size=128]
    Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
    Status: D0 PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/0 Enable+
    Address: 00000000fee0300c Data: 4169
    Capabilities: [58] Express Endpoint IRQ 0
    Device: Supported: MaxPayload 512 bytes, PhantFunc 0, ExtTag-
    Device: Latency L0s unlimited, L1 unlimited
    Device: AtnBtn- AtnInd- PwrInd-
    Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
    Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
    Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
    Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 0
    Link: Latency L0s unlimited, L1 unlimited
    Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
    Link: Speed 2.5Gb/s, Width x1
    Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
In this specific case, the device supports 512 bytes and it's programmed to assume a maximal payload of 128 bytes. This is the common situation: The device's max payload size is larger than is effective (in fact, it seems like 128 bytes is used everywhere).

Earlier in the same lspci -vv there's the entry for the relevant root port (found with lspci -t):

    00:01.0 PCI bridge: Intel Corporation 82G33/G31/P35/P31 Express PCI Express Root Port (rev 10) (prog-if 00 [Normal decode])
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
    Latency: 0, Cache Line Size: 4 bytes
    Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
    I/O behind bridge: 0000a000-0000afff
    Memory behind bridge: fda00000-fdafffff
    Prefetchable memory behind bridge: 00000000fd600000-00000000fd600000
    Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
    BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
    Capabilities: [88] #0d [0000]
    Capabilities: [80] Power Management version 3
    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
    Status: D0 PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [90] Message Signalled Interrupts: 64bit- Queue=0/0 Enable-
    Address: 00000000 Data: 0000
    Capabilities: [a0] Express Root Port (Slot+) IRQ 0
    Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
    Device: Latency L0s <64ns, L1 <1us
    Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
    Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
    Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
    Link: Supported Speed 2.5Gb/s, Width x16, ASPM L0s, Port 2
    Link: Latency L0s <1us, L1 <64us
    Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
    Link: Speed 2.5Gb/s, Width x1
    Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug- Surpise-
    Slot: Number 32, PowerLimit 10.000000
    Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
    Slot: AttnInd Off, PwrInd On, Power-
    Root: Correctable- Non-Fatal- Fatal- PME-
    Capabilities: [100] Virtual Channel
    Capabilities: [140] Unknown (5)

Here we see that the root port is limited to 128 bytes, which is consistent with the setting of the device.

I hope this clarified a bit.
Eli
support
 
Posts: 619
Joined: Tue Apr 24, 2012 3:46 pm

Re: PCIe Max Payload Size

Postby Guest » Wed May 08, 2013 9:21 pm

Thank you for the reply back. How can I change the value? e.g., 128byte to 4096byte?
Guest
 

Re: PCIe Max Payload Size

Postby support » Wed May 08, 2013 9:27 pm

Whether you can convince the root port to work with a larger maximal payload size is a question of whether the hardware (e.g. bus chipset) supports it. But odds are you can't.

You may write directly to the device's configuration space and alter the assigned maximal payload size, but it would send packets that are illegal on the bus as a result of that -- so I can't see the good in doing that.
support
 
Posts: 619
Joined: Tue Apr 24, 2012 3:46 pm

Re: PCIe Max Payload Size

Postby Guest » Wed May 08, 2013 10:16 pm

Oh I see, thank you so much for the explanation below. I already explored the system BIOS and found one system that had two options, 128 and 256byte. I was hoping to set the BIOS feature to 4096 for a max efficiency per transfer but most of the system BIOS default is set to 128bytes. Then I looked into the OS side, hoping there's a feature or tool that can change the RC value to match EP but so far, I haven't found anything that will do the job. e.g., if RC = 128bytes and EP 215bytes, RC < EP. How can the RC scan and adjust the MPS value to 215 during a boot up?
Guest
 

Re: PCIe Max Payload Size

Postby support » Wed May 08, 2013 10:23 pm

How the BIOS can update the root port's MPS is hardware dependent. I suppose it's some register.

Anyhow, the BIOS picks the RC's MPS first, and then scans the bus. The effective MPS is the smallest of the RC's and each of the devices (typically, they all get the RC's MPS)
support
 
Posts: 619
Joined: Tue Apr 24, 2012 3:46 pm

Re: PCIe Max Payload Size

Postby Guest » Wed May 08, 2013 10:36 pm

Yes, you are right. I guess there isn't much that can be done on the OS side then. Do you know why some EPs have 215bytes or larger MPS when they should know that most of the system BIOS limit MPS to 128bytes?
Guest
 

Re: PCIe Max Payload Size

Postby support » Wed May 08, 2013 10:39 pm

I can only guess that endpoints are designed to work well with future hardware. The interesting question is why the RCs don't increase their MPS. Which I can only speculate about as well.
support
 
Posts: 619
Joined: Tue Apr 24, 2012 3:46 pm

Re: PCIe Max Payload Size

Postby Guest » Wed May 08, 2013 10:49 pm

Yes, I'm a little puzzled by it as well. Again, thank you so much for your explanation.
BTW, I love your blog, pls keep it up~ =)
Guest
 


Return to General PCIe

cron