Read and write for Bus Mastering

Comments and questions related to the "Down to the TLP" pages

Read and write for Bus Mastering

Postby Guest » Fri Feb 07, 2014 7:01 pm

Can you please explain how the read and write to the CPU memory by the peripheral happens when Bus Mastering is enabled ?
Assuming I have a WLAN card as the peripheral, how will we transmit and receive packets over the air ?

Re: Read and write for Bus Mastering

Postby support » Fri Feb 07, 2014 8:49 pm

OK, this is going to be a bit of a story, but let's go for it. The details of how the communication takes place differs from one hardware model to another (which is why different drivers are necessary), but the principles tend to be the same.

I'll assume a plain network card (NIC) for simplicity. And this is really a skeleton outline of what happens.

To begin with, the driver makes a series of writes to the NIC's registers. To the processor, these are plain memory writes (as in *pointer = data) but to addresses that are typically allocated to the PCIe BAR region. So effectively, each such write operation causes a write TLP to be generated by the processor's PCIe switch, and sent through the bus.

By writing to these registers, the driver software configures the hardware, setting control registers to values that have some meaning to the hardware. For example, if it should negotiate the link speed, maybe change its MAC address etc. The meaning of these registers depends on the hardware.

The driver also allocates buffers for DMA transmission in its own RAM. The addresses of these buffers are also written to some dedicated registers. For simplicity, let's assume that there's one buffer for received packets, and one for transmitted packets.

Now suppose that a packet arrives from the network to the NIC. The NIC reacts by writing the packet's data into the DMA buffer -- it knows its address from one of its registers. From a PCIe point of view, the NIC creates a write TLP (or several such) requesting to write the data in the packet's payload at a given address (the DMA buffer's address in our case). Note that the NIC accesses the host's memory space directly, hence it's Direct Memory Access (DMA). Or one could say that the NIC functions as a bus master by sending these packets. Both expressions mean the same here.

When the NIC has finished sending the TLPs for writing, it needs to tell the CPU that a network packet has arrived, usually by issuing an interrupt. Typically, this will be dome with an MSI, which merely consists of writing a certain value to a certain address address. Or, in PCIe terms, sending a write TLP with a specific required address and a specific value in the payload.

The typical CPU response to such an interrupt is to read some status register from the NIC, to sort out what the interrupt was about, and then consume the data in the DMA buffer. And finally, tell the NIC that the buffer's data has been consumed, and that it can be reused, by writing to a register.

Now, if the CPU wants to send a network packet, it fills the DMA buffer with its data, and writes to a register that tells the NIC that there's a network packet ready for transmission. The NIC responds with sending a read request TLP to the CPU (or several such), with the address field in these packets pointing at the DMA buffer. The CPU's PCIe bus infrastructure fetches the data from the CPU's RAM, and sends the data back to the NIC as Completion TLPs. Note that the software running on the CPU is completely unaware of this taking place -- the processor is not involved in this packet exchange. This all happens among bus switches, bridges and the memory controller.

When the last completion packet has arrived to the NIC, it sends and interrupt to the CPU to inform it that the data has been consumed, and the buffer may be reused.

That's more or less what happens. In a real-life NIC, there are many DMA buffers allocated at once, and several other optimizations are made. But the principle of any bus master enabled peripheral is more or less like this.

I hope this clarified things.

Posts: 760
Joined: Tue Apr 24, 2012 3:46 pm

Re: Read and write for Bus Mastering

Postby Guest » Mon Feb 10, 2014 10:54 am

Thank you Eli for the explanation.

Return to General PCIe