PCIe Payload size limit

[Guest]

Hi there,
Im working on an fpga pci device, and i noticed that reading is faster than writing.
when looking at lspci -vv

i see that my kernel is giving my device only 256 MaxPayload, I understand that the kernel does this according to the slowest device on the path between my fpga to the root complex.
what can i do to figure out what device is limiting my payload size?and how can i fix this?

Thanks alot

[support]

Hi,

You just have to ask me.

Odds are that you have no PCIe switch between the computer and the device (and probably, it wouldn't matter so much anyhow) so there's only the device and the root port involved. So the reason your Maximal Payload is 256 bytes is your computer, basically.

In fact, until quite recently, the common figure was 128 bytes, but newer motherboard have improved.

This way or another, there isn't much you can do about it. I'm not sure if there is any hardware with 512 bytes out there. Or if it's worth bothering looking for one.

Now let's get back to the beginning of your note: Reading faster than writing. What were the figures you measured? Was it host-initiated I/O or DMA?

Or maybe it isn't that important to go down that road. The Maximal Payload rules for both read and write payloads. So I pretty much doubt it's to blame anyhow.

Regards,
Eli

[Guest]

Hi thanks,
Im using the fpga as a "storage device". Im using the dram on it for the storage.
The device is controlled by a block driver the issues reads and write.
When the driver issues a read, the device sends the packets to the correct address and the maxpaylad is 256B.
When the driver issues a write, the device starts sending requests to the memory and get back tlp's with 512B.
Is there really nothing i can do to improve my reads?

Thanks

[support]

Hi,

Even with these figures, the header vs. payload overhead is rather small. What results do you have, and how much would you like to improve them?

Regards,
Eli

[Guest]

Its hard to talk about the results of only the pci without the block layer and driver and implementation of the device.
But for example when i run "dd bs=1M count=1000 of=/dev/zero if=/dev/vdiskb_a iflag=dsync,direct" (read from device)
I get about 700MB/s
While running "dd bs=1M count=1000 if=/dev/zero of=/dev/vdiskb_a oflag=dsync,direct"
I get about 1G/s

The read and write functions of my driver do more or less the same so i want to understand why are the results so different

[support]

Hello,

I can't really explain this either, but surely a 30%-40% difference can't be a result of maximal payload size. No way. Had it been a few percents, it would have made sense.

The only thing I can speculate is that maybe data is stored in some RAM buffers that are related to the block memory (or disk cache) on the write operation, despite the flags used. Does this difference remain on a test that runs for, say, a minute? A test like that would cancel any buffering effects.

Regards,
Eli