BAR0/1 & enumeration (NTB)

Comments and questions related to the "Down to the TLP" pages

BAR0/1 & enumeration (NTB)

Postby Guest »

Eli, I'm one of the thousands who browse and read your blog silently.

But now I need to get in touch about something I hope you can help. Here it is.

I'm working on a NTB driver. NT functions have Type-0 headers, like any PCIe end point.

Now, BAR0/1 on NT functions can be setup (in general via eeprom) as base address for the 4K config-space of its function OR as based address of the memory space for memory transactions. In any case, during enumeration the bios must write the proper base address into BAR0/1.

My question is:
How can the bios during enumeration know whether BAR0/1 is setup for pointing to its function config-space or to memory space for memory transactions?

Hope you can help.
Kind Regards
Guest
 

Re: BAR0/1 & enumeration

Postby support »

Hi,

I'm not an expert on NTB, but I think I spot a slight confusion here: In PCIe terms, "configuration space" is where the device's presence on the bus is configured. There is no address to set up about it, because it has to be present when the system boots up. As a matter of fact, this is where the BIOS reads the desired BAR allocations from and allocates the BAR addresses to.

So obviously, the only thing the BIOS does is to set up memory access addresses for the BAR entries (or I/O addresses on legacy systems).

On the other hand, the NTB needs a BAR segment to control its behavior, e.g. sending doorbells to the other side. This is (maybe) also called "configuration" but it's actually regular memory mapped BAR registers. Another BAR segment is used as a "gateway" to the other side, and would be called "memory space". So an NTB appears to be just another device to the BIOS, with at least two BAR segments.

Like any device, the driver needs to know which BAR means what.

The real catch is if the NTB's infrastructure (i.e. the PCIe lanes) may be used as a root port. In that case, the BIOS needs to be aware of the user's choice at bootup, and set some register accordingly. This is implementation-specific.
support
 
Posts: 802
Joined:

Re: BAR0/1 & enumeration

Postby Guest »

Eli, thanks a lot. A quick reply:

>> There is no address to set up about it,

Right. What I meant was that the 4K configuration space for each BDF gets also memory mapped (by the BIOS?) starting from 0xE000.0000.

>> On the other hand, the NTB needs a BAR segment to control its behavior, e.g. sending doorbells to the other side. This is (maybe) also called "configuration" but it's actually regular memory mapped BAR registers.

This shed light. The message & doorbell registers are within the 4K config space of the NT function. You are saying that a BAR may be used for memory-mapping the config-space such that the other side "knows" where the registers of this side are. And vice-versa.

Makes sense but, why mapping it AGAIN? It's already mapped as I said above.

>> The real catch is if the NTB's infrastructure (i.e. the PCIe lanes) may be used as a root port. In that case, the BIOS needs to be aware of the user's choice at bootup, and set some register accordingly. This is implementation-specific.

I don't think the BIOS needs to know. In fact, the NT is just another function for that same root port. The NTB transfers data via the NT function. The transparent bridge transfers data via the regular P2P function.

Ok, I have to figure out why mapping twice the NT config space.

Best Regards.
Paolo
Guest
 

Re: BAR0/1 & enumeration

Postby support »

Guest wrote:>> On the other hand, the NTB needs a BAR segment to control its behavior, e.g. sending doorbells to the other side. This is (maybe) also called "configuration" but it's actually regular memory mapped BAR registers.

This shed light. The message & doorbell registers are within the 4K config space of the NT function. You are saying that a BAR may be used for memory-mapping the config-space such that the other side "knows" where the registers of this side are. And vice-versa.

No, I'm actually saying something much simpler. The host needs to access these e.g. doorbells control registers somehow, so they have to be registers mapped on the host's memory map. So it's just a plain BAR, mapped by the BIOS like any peripheral's registers. Aside from that, there's this gateway region, which is another BAR, which is subject to traversing the bridge and having the addresses translated.

Guest wrote:>> The real catch is if the NTB's infrastructure (i.e. the PCIe lanes) may be used as a root port. In that case, the BIOS needs to be aware of the user's choice at bootup, and set some register accordingly. This is implementation-specific.

I don't think the BIOS needs to know. In fact, the NT is just another function for that same root port. The NTB transfers data via the NT function. The transparent bridge transfers data via the regular P2P function.

If the infrastructure is used as an NTB, then the outer world expects to see a bus device. If no NTB is used, why waste the hardware lines, when they can be used as a root port to connect other devices to it? In that case, the BIOS needs to enumerate whatever is attached to these lines. Hence the BIOS has to know at bootup what the choice was. But this is really not the topic here.

Eli
support
 
Posts: 802
Joined:


Return to General PCIe