board to board

Comments and questions related to the "Down to the TLP" pages

board to board

Postby Guest » Tue Jul 21, 2015 5:50 pm

Hi Eli,

Firstly, thank you for taking the time to write the "Down to the TLP" blogs. They've been extremely helpful in understanding how it all works.

I have two PCIe cards, each with Altera FPGAs on them with their PCIe HIP / DMA reference design in them. I can DMA to and from each device perfectly. Both devices (let's call them device1 and device2) have the DMA controller(s) in BAR 0 and 4K of SRAM in BAR 2. From the host, I can DMA to and from the SRAM (to/from host memory). I verify the data by inspecting BAR 2. I then tried to DMA from device1's SRAM to the physical address of device2's bar2 mapping. From what I've read, this should be legal.. ? It actually works as long as the dma length is one single 32-bit word. If i increase it, even to 2, it hangs the system. Can you think of any reasons why this might not work? The DMA turns into a write (from device1) to host physical memory (so the ID would be for the host, not for device 2), I assume? Is that the problem? Any insight greatly appreciated! Thank you!

Re: board to board

Postby support » Tue Jul 21, 2015 7:02 pm


You seem to be trying peer-to-peer DMA, which is well within PCIe's specification. So at least in theory, this should work: The root port (probably hosted by the CPU chip or its memory controller) should route any packet that is mapped to a BAR range to the BAR's owner, regardless of its origin: The CPU or any device on the bus. Same goes for any PCIe switch on the way (if there is any).

I have no personal experience with this as of yet, but it's a rather common practice. Why it failed in your case -- it's not clear. Maybe the packets are malformed to begin with, and the root port was forgiving enough when they were destined to the CPU's memory. And maybe your host doesn't support peer-to-peer. Sometimes CPUs don't really support the full PCIe spec (or have bugs regarding less used features), in particular old (like three years) x86 and certain embedded devices.

if you don't have the freedom to choose a suitable root port, you may want to consider adding your own switch. For example, an evaluation board from Avago (formerly PLX) could be helpful.

Posts: 760
Joined: Tue Apr 24, 2012 3:46 pm

Return to General PCIe