64bit usage of PCIe driver

Questions and discussions about the Xillybus IP core and drivers

64bit usage of PCIe driver

Postby tozkoparan »

Hello

We tried demo applications given in the link: http://xillybus.com/downloads/xillybus-windowspack.zip and realized that streamwriter.c application cannot send files over 1 gigabyte.

First; we thought that 32 bit compiled streamwrite.c applications have 1 gigabyte constraint because of malloc or file system buffer usage restrictions.

Then we compiled streamwrite.c application as 64 bit, but we see the same constraint again.

So we suspect that pci driver does not support buffer transfers over 1 gigabyte.

İs our conclusion right?

Best Regards.
tozkoparan
 
Posts: 15
Joined:

Re: 64bit usage of PCIe driver

Postby support »

Hello,

No, your conclusion is heavily wrong. There is no restriction whatsoever on the amount of data sent over a Xillybus stream. Not the IP core and not the driver.

Why it stopped on 1 GB remains to find out. What exactly did you do? For example, did you untie the loopback? Or did you consume the data on the other side?

Regards,
Eli
support
 
Posts: 802
Joined:

Re: 64bit usage of PCIe driver

Postby redalert »

Our aim is to send large data blocks such as 64GB to FPGA through Xillybus . According to our requirements, sent stream should have continuity without gaps. Our system use 64 bit Windows 7 OS. So you can consider our questions according to Windows OS. Our windows compiler settings and malloc usages are arranged according to 64 bit.

But we need to solve some problems to realize our requirement.

1) _write call cannot get buffer size over 2GB. At most, it is value can be assigned to 0x7FFFFFFF. If we set the sent write buffer size to 2GB ( 0x80000000), then write call immediately returns with -1. It seems that there is a restriction for the write call buffer size limit for 2GB. It is understandable since Windows Kernel separates 1GB at most. Can you confirm that restriction ?

2) Because of the condition about 2GB, we cannot set the "len argument" of write call buffer as 64 GB. Since len-sent will be over 2GB and write call return -1 immidiately, and send operation cannot be realized.

void allwrite(int fd, unsigned char *buf, int len) {
int sent = 0;
int rc;

while (sent < len) {
rc = _write(fd, buf + sent, len - sent);

if ((rc < 0) && (errno == EINTR))
continue;

if (rc < 0) {
perror("allwrite() failed to write");
exit(1);
}

if (rc == 0) {
fprintf(stderr, "Reached write EOF (?!)\n");
exit(1);
}

sent += rc;
}
}

(Note: we have adapted len , sent and rc arguments for 64 bit by changing types from int to size_t)

Instead of directly giving (len-sent) , we modified code to sent buffers consecutively having maximum size 0x7FFFFFFF.

But in spite of we intended to realize consecutive write calls, each write call was blocked by transmission.
During transmission, write call cannot return and we cannot feed consecutive buffer.
Hence we experienced that write calls are blocked during the time interval till completion of transmission.

That problem cause to gaps in out transmission time. We need circular buffer like driver api that provide buffer feeds during transmission time.
And consecutively sending queued buffers without any gaps.

We reviewed windows developer manual and demo apps codes. But we cannot point to the solution.

Can you inform us about this conflict. Is it possible to realize our scenario with your api.

thanks for your support.

Best Regards.
redalert
 
Posts: 2
Joined:

Re: 64bit usage of PCIe driver

Postby support »

Hello,

Xillybus is intended for continuous transmission of data, with no gaps. Xillybus' DMA buffers are in place to ensure that the data flow continues between calls to _write().

If you have data gaps, it's probably because the application stalls (most often reading data from the disk), or just something done wrong.

There is no need to call _write() with huge numbers for the buffer length. Even with write requests as small as 128 kB, the system call overhead is pretty negligible. So instead of (len - sent) as the third argument to _write, the minimum of (len - sent) and 128k would most likely have done the job. That works around the API's limitation.

So let's have this clear: You have a 64 GB buffer which is resident in memory (pinned, can't be swapped into disk), and you're using allwrite() to write from it? No disk or other slower device involved?

What is your required data rate?

Regards,
Eli
support
 
Posts: 802
Joined:

Re: 64bit usage of PCIe driver

Postby redalert »

At our previous trials, different buffer size selections failed. But 128KB works.
Thank you very much.

Best Regards,
redalert
 
Posts: 2
Joined:

Re: 64bit usage of PCIe driver

Postby Guest »

Hi, thanks again for the previous question's answer. Another issue we have, that your experiences may lead us.

In our case there exist 1 Tx Thread and 4 Rx Threads. I mean with Tx as HOST to FPGA and RX: FPGA to HOST direction.

FPGA Tx and Rx Configurations are asynchronous.

We use Fifo.c demo app as base for RX Threads.

As you know, Fifo.c have three threads:

1) write_thread

2) read_thread

3) status_thread

read_thread _read call get buffers from dma and write_thread writes to disk. We use SSD as disk.

We successfully run RX and TX threads separately. Tx stream is continuous without gap.

When we run RX and TX threads at the same time, we observe that tx streams have gaps.

We suspect that RX threads schedule burden cause to slow down tx threads _write call loop.

We solved this issue with "UGLY" method by putting sleep to rx thread (read_thread in fifo.c i.e. after _read call).

With this "UGLY" case, 4*RX and TX threads runs successfully and TX stream is continuous without gap.

For asynchronous case, do you have any recommendation of the _read call usage on RX threads or _write call usage on TX thread, instead of our "UGLY" solution with sleep.
Guest
 

Re: 64bit usage of PCIe driver

Postby support »

Hello,

I don't have a clear picture of what you're doing there (data rates?), but usually when adding sleep() just somewhere helps, it has to do with caching: The processor's cache, disk cache. Or I/O scheduling (in particular with SSDs). I can't say anything more specific, unfortunately.

Regards,
Eli
support
 
Posts: 802
Joined:


Return to Xillybus