PCIe 2.0 and lane size

Comments and questions related to the "Down to the TLP" pages

PCIe 2.0 and lane size

Postby Guest »

Hi! Maybe it is wrong place to ask but I'll still try :).

I wrote a (DMA) linux driver for Xilinx deviced (AXI-PCI core) and tested it with zc706 and vc707 devices. It works fine, but it looks like that the speed is limited to PCIe Gen1 4lane and I have absolutely no idea why. It never goes above 800 MB/s. I can only slow down it in half by changing the switch position on the board.
Lane of course might be broken, but I don't undersand what is wrong with Gen1/Gen2 thing. I tried it on motherboards with PCIe Gen2 support and on boards with Gen1 only. No difference. I think that I miss something on the kernel side or on the fpga side. It looks like that I need to switch it somewhere somehow. In AXI-PCI core settings there is only one tab which seems related: the one, where you change number of lanes and the link speed. Changing these setting actually change absolutely nothing for me. It is always around 800MB/s or slower.
Guest
 

Re: PCIe 2.0 and lane size

Postby support »

Hello,

What does lspci -vv (run as root) tell you about the device?

For example, my VGA card gives this:

Code: Select all
01:00.0 VGA compatible controller: ATI Technologies Inc RV710 [Radeon HD 4350] (prog-if 00 [VGA controller])
        Subsystem: Giga-byte Technology Device 21ac
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 48
        Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Region 2: Memory at fbce0000 (64-bit, non-prefetchable) [size=64K]
        Region 4: I/O ports at ae00 [size=256]
        [virtual] Expansion ROM at fbc00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 00000000fee8800c  Data: 4172
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: radeon
        Kernel modules: radeon


Note the LnkCap and LnkSta entries: The first says what the device reports that it's capable of, and the second what was actually negotiated.

The terminology of different versions of lspci varies.

Regards,
Eli
support
 
Posts: 802
Joined:

Re: PCIe 2.0 and lane size

Postby Guest »

I tried it with vc707. When PCIE-AXI core configured in 4x mode it says what it should. 5GT/s when it's configured as 5GT/s and 2.5GT/s when it is 2.5GT/s.
When I transfer the data it is about 800MB/s for 2.5GT/s (approximately what it should be) and for 5GT/s it looks like it slightly faster (about 900MB/s). On FPGA side there is AXI-PCIE core and CDMA core which does all the data transfer. Might it be because of the PCIe extender cable?

With 8x there is another story. When I configure AXI-PCIE as 8x 2.5GT/s (doesn't support 5GT/s) LnkCap is 8x 2.5GT/s, but LnkSta is 4x 2.5GT/s. Can't say where is the problem. PCIe slot on the motherboard is PCIe Gen2 16x. J49 on FPGA jumper is in 8x mode.
Guest
 

Re: PCIe 2.0 and lane size

Postby support »

Hello,

The only thing that looks fishy to me is the 900 MB/s rate at 4x Gen2. You should see about 1600 MB/s at least. As for the extender cable, you can see if there are hardware level problems quite easily (I wouldn't bet my money on that direction):

http://billauer.co.il/blog/2011/07/pcie ... yer-error/

Unfortunately, I don't have any experience with neither the AXI-PCIe bridge, nor the CDMA core. I suggest taking this to Xilinx' forum.

Regards,
Eli
support
 
Posts: 802
Joined:


Return to General PCIe

cron