Thank you for the nice and neat PCIe article.
I am working with PCIe and I'm using mmap to be able to directly write and read. Everything works fine, however my read latency is awful. I needed to understand what's happening in underlying layers and this article gave me a basic understanding to start with. I suppose the non-posted nature of read is causing the long delay. You mention that this is "rightfully avoided" in modern systems, but I don't understand what you exactly mean by that. Can anyone help me understand or find out what's going on in the underlying layers of Linux kernels, and if there is anything I can do to optimize it? Here is a simple piece of code that shows how to use mmap to communicate with a PCIe device in Linux.
- Code: Select all
int fd = open("/dev/mem", O_RDWR|O_SYNC);
int pci_bar0 = 0x0a0b0c0d; // This is the PCI device base address you can find with lspci or /proc/bus/pci/devices
char* ptr = mmap(0, MMAP_SIZE, PROT_READ|PROT_WRITE, MAP_SHARED, fd, pci_bar0);
if(ptr == MAP_FAILED)
perror("MMAP FAILED\n");
else
printf("PCI BAR0 0x0000 = 0x%4x\n", ptr);
// Now you can read or write like a simple pointer. Of course, you should know the address map of your device
*(ptr+offset) = something-to-write;
read-something = *(ptr+offset);