PCIe -- follow up questions

Comments and questions related to the "Down to the TLP" pages

PCIe -- follow up questions

Postby Guest » Sun Sep 28, 2014 4:51 pm

Thank you very much for your three articles introducing PCIe. I have two questions:

Question 1 on why transfer rate as a function of amount of data transferred continues to improve even after going beyond maximum payload size: I understand the payload size request in the configuration register can be from 128 bytes to 4096 bytes in powers of two, and that the root complex informs all nodes of the maximum data payload (MDP). Suppose in some system the MDP happens to be 4 kB. If one plots the data transfer rate for various sizes, why doesn't the curve level off after 4 kB? Because even if 1000*4 kB needs to be transferred, it will be transferred as 1000 separate transfers of 4 kB each. So the question is: even though payloads have a maximum size of 4 kB, why do we see improved data transfer rates for transferring amounts larger than 4 kB?

Question 2 regarding "No end-to-end acknowledgment is ever made, and neither is it really necessary.": Consider the need for A to write to B and inform C when write has completed. C needs to know when all the data has arrived at B and B has "flushed its FIFOs" to DDR, and not just when A has finished sending out data. If A sends to B a 0-byte read from the very last write-address, and gets back the 0-byte data, is it guaranteed that B has flushed all data to DDR and so now is the right time for A to sent a MSI to C informing it of completion of the write?

Thanks.
Guest
 

Re: PCIe -- follow up questions

Postby support » Sun Sep 28, 2014 6:48 pm

Hello,

Question 1 on why transfer rate as a function of amount of data transferred continues to improve even after going beyond maximum payload size: I understand the payload size request in the configuration register can be from 128 bytes to 4096 bytes in powers of two, and that the root complex informs all nodes of the maximum data payload (MDP). Suppose in some system the MDP happens to be 4 kB. If one plots the data transfer rate for various sizes, why doesn't the curve level off after 4 kB? Because even if 1000*4 kB needs to be transferred, it will be transferred as 1000 separate transfers of 4 kB each. So the question is: even though payloads have a maximum size of 4 kB, why do we see improved data transfer rates for transferring amounts larger than 4 kB?


On almost every system out there, the maximal payload is 128 bytes, maybe 256 bytes. Why your efficiency increases as you send more data depends on your specific setup, and how you measure performance. So I can't say much on this.

Question 2 regarding "No end-to-end acknowledgment is ever made, and neither is it really necessary.": Consider the need for A to write to B and inform C when write has completed. C needs to know when all the data has arrived at B and B has "flushed its FIFOs" to DDR, and not just when A has finished sending out data. If A sends to B a 0-byte read from the very last write-address, and gets back the 0-byte data, is it guaranteed that B has flushed all data to DDR and so now is the right time for A to sent a MSI to C informing it of completion of the write?

The only thing a zero-byte read gives you, is that the request for reading will appear at the endpoint after the write. For example, if you have a TLP interface in an FPGA, the TLP for the read request will arrive after the TLP for write request. This, and only this, is guaranteed.

If you want to use this mechanism for synchronizing a flush with the host, make sure that the completion for the read request is sent only after flushing the write data. This is however not a very good practice, as there is a PCIe timeout for read requests, and you don't want to get there. Besides, the host's bus may become locked, freezing the processor completely until the read's completion arrives.

It all boils down to how much time the flush is expected to take, and how often these synchronizations are necessary. But I would definitely go for an interrupt if it's an occasional need.

Regards,
Eli
support
 
Posts: 760
Joined: Tue Apr 24, 2012 3:46 pm


Return to General PCIe

cron