The Xillybus Forum

Hello,

I have a problem with Xillybus on a Raspberry Pi 5 with Linux kernel 6.1.0. When insmod'ing the driver, I get a kernel oops from swiotlb_map.

The message of this oops is:

xillybus_pcie 0000:01:00.0: swiotlb addr 0x00000010fdffc000+16384 overflow (mask ffffffff, bus limit 1fffffffff).

Then there is a log message saying "xillybus_pcie 0000:01:00.0: Failed to map DMA memory!. Aborting."

There are no device files. Please help!

Hello,

There is something inherently wrong with the kernel's memory allocation and/or DMA handling. There is no reason for the SWIOTLB mechanism to get involved.

It's possible to work around this problem by forcing the driver to work with 64-bit DMA: Edit this part in xillybus_pcie.c:

Code: Select all

    if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(32)))
        endpoint->dma_using_dac = 0;
    else if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(64)))
         endpoint->dma_using_dac = 1;
     else {
         dev_err(endpoint->dev, "Failed to set DMA mask. Aborting.\n");
 
     [ ... ]


And change it to:

Code: Select all

    if (!pci_set_dma_mask(pdev, DMA_BIT_MASK(64)))
         endpoint->dma_using_dac = 1;
     else {
         dev_err(endpoint->dev, "Failed to set DMA mask. Aborting.\n");

        [ ... ]


It's clear from the oops message that 32-bit DMA was in effect. The kernel should probably have refused to this, and let the driver fall back to 64 bits. With this change, 32-bit DMA isn't an option, so this solves the problem with DMA mask. This way, the kernel will have no excuse to mess around with SWIOTLB.

I also suggest changing the memory allocation routine, so it doesn't limit the DMA buffers to 32 bits, as it currently does. Edit xillybus_core.c, and get rid of the __GFP_DMA32 flag.

In other words, change from this:

Code: Select all

    addr =  __get_free_pages(GFP_KERNEL | __GFP_DMA32 | __GFP_ZERO, order);

To this:

Code: Select all

    addr =  __get_free_pages(GFP_KERNEL | __GFP_ZERO, order);

This probably won't matter, because the kernel appears to have ignored the flag either way.

It's best to use IP cores that have been generated after July 2021 (and demo bundles downloaded after this date) along with 64-bit DMA. These IP cores are handling 64-bit DMA better than those previously released.

For general information about SWIOTLB, see this page:

https://xillybus.com/tutorials/iommu-swiotlb-linux

Regards,
Eli