Trouble getting x4 lanes working

Questions and discussions about the Xillybus IP core and drivers

Trouble getting x4 lanes working

Postby Guest » Thu Aug 30, 2018 4:11 pm

Eli, thanks for helping with my prior issues. Now I'm able to actually build good bitstreams and test them. However, the PCIe bus isn't working properly.

  • I've previously gotten the xillybus demo fully working on the KC705
  • My target board PCIe is 4-lane, not 8-lane
  • I've gotten the Xilinx PCIe demo working on my board at 4 lanes. This includes moving pcie lane, pcie clock, and pcie reset signals to non-standard pin locations used by the custom board.
    • Note that I have the working Xilinx PCIe demo in my config flash. I must cold start (with jtag usb cable removed from PC), in order for BIOS to enumerate PCIe bus. After that, I can jtag/config a new bitstream and simply warm boot my linux host.
    • I have a VIO in there. It shows me the link up signal. (The Xilinx PCIe demo has an active high link up signal, while the Xillybus demo has an active low link up signal.)
    • I can re-jtag/re-config the same Xilinx PCIe demo bitstream. At first, the VIO shows the link is NOT up. This is expected because I momentarily broke the PCIe connection with the host. I warm start the linux host. The VIO immediately shows the link IS up. A minute later I can log in to linux host and lspci shows my custom board. Remember, this is using the Xilinx PCIe demo bitstream. (No driver yet.)
  • I am having trouble getting the xillybus demo fully working on my custom board at 4 lanes.
    • I made what I believed to be the correct changes to go down from 8 to 4 lanes. This included editing xillybus.v (now xillybus_4x.v), pcie_k7_8x.v (now pcie_k7_4x.v), and reconfiguring ip pcie_k7_vivado (now pcie_k7_vivado_4x). It also included moving those pin locations.
      • FAILED as follows
      • When I jtag/config this xillydemo bitstream, the VIO shows me that the link is NOT up, as expected. Then I warm boot my linux host. The VIO very quickly shows me that the link is up, as expected and so far so good.
      • However, the linux host won't boot further. It's hung somehow. I believe it's hung in the BIOS, but I'm not certain. So the VIO says the link is up, but the system linux host boot correctly.
    • So now I try replacing the lower level portions of the xillydemo with the lower level portions of the Xilinx PCIe demo. Maybe there's a snafu in something I did. I'm copying low level modules from a working project to a non-working project. I replaced the actual Xilinx PCIe IP (pcie_k7_vivado_4x.xci replaced by pcie_7x_0.xci) and also the pipe_clock module (pcie_k7_vivado_pipe_clock.v replaced by pcie_7x_0_pipe_clock.v).
      • FAILS in the same way
      • It seems like the only difference in the two projects now is at the FPGA "application" level. That is, the Xilinx PCIe demo is using a Xilinx-provided pcie_app_7x.v, while the modified xillydemo is using xillybus_core.v as well as top level logic.
      • I wondered if the xillybus_core needed to be different for x4 lane operation instead of x8 lane operation, but I see no such setting in the IP Factory. (This is Kintex-7, of course, just like KC705.)
      • Note that when I replaced pcie_k7_vivado_pipe_clock.v with pcie_7x_0_pipe_clock.v, I saw differences in the parameters provided. PCIE_LINK_SPEED changed from 3 to 2. PCIE_USERCLK1_FREQ and PCIE_USERCLK2_FREQ want to change as well, but I left them at the xillydemo values of 4 because the Xilinx PCIe demo code was a bit obtuse and I didn't want to get it wrong.

Your suggestions are GREATLY appreciated!
Thanks,
Helmut
Guest
 

Re: Trouble getting x4 lanes working

Postby Guest » Thu Aug 30, 2018 4:53 pm

  • I find that I can take my working Xilinx PCIe demo and totally comment out the "app", and the bitstream still builds. This leads to the SAME linux host boot hang as I've seen with the Xillybus demo (with VIO showing link up).
  • The above "proves" that the "application" connecting to the Xilinx PCIe lower levels is "important" in allowing the linux host to boot all the way.
  • I find that I can take my failing Xillybus demo (with all PCIe stuff replaced with code from working Xilinx PCIe demo) and also totally comment out the xillybus_core, and the bitstream still builds. As expected, I still get the linux host boot hang (with VIO showing link up).

This leads me to the strong belief that my xillybus_core is simply good for 8-lanes and NOT good for 4-lanes.

I don't see how I can specify this in the IP Factory. Is this a case where a custom xillybus_core is required?
Guest
 

Re: Trouble getting x4 lanes working

Postby support » Thu Aug 30, 2018 6:49 pm

Hello,

I have to admit that I got somewhat lost in the rather lengthy explanation above. In particular I didn't understand what you meant with Linux not booting further. What exactly happened there?

I'll therefore point out some general points that may be helpful:

* Xillybus IP core is agnostic to the underlying PCIe block's parameter, as long as the links works properly.
* In particular, a 4-lane bundle for the Kintex family is available at request -- it's the bundle for ZC706. It so happens that the Zynq device involved is just a Kintex 7 with an ARM, and the PCIe finger on the board is just 4 lanes wide. So exactly the same IP core is used, but the bundle for the board is different. You may request it by email. Needless to say, it works.
* The FPGA must be configured when the PC goes out of reset. According to Xilinx, if the FPGA board is powered by the PC and configured from flash it has something like 100 ms for the configuration. Quite often, this rule can be broken, and still things work fine. Or wiggly. Or fail for random reasons. Your setup isn't quite clear to me, so I don't know if this is relevant.

Regards,
Eli
support
 
Posts: 623
Joined: Tue Apr 24, 2012 3:46 pm

Re: Trouble getting x4 lanes working

Postby Guest » Thu Aug 30, 2018 7:26 pm

Eli,

Thanks. I'll try to clarify and respond...

* When I jtag config (not boot from config flash) with the x4-lane-modified Xillydemo, the host shows some BIOS info on the monitor. Under other circumstances, this BIOS info is followed by linux boot info. But in this case, after the BIOS info comes nothing. Shortly thereafter the monitor powers off for lack of incoming signal. I believe either the BIOS is never finishing or linux boot is never getting past the very early stages. I suspect BIOS for this reason. When boot order was to boot from a PCIe/104 nic first, the BIOS message trying to boot from the nic was stuck showing. The BIOS was stuck. When boot order was to boot from disk first, we got the monitor power off as described. My theory was that the PCIe bus was getting [somehow hung up, no specific knowledge] and this made the BIOS get stuck trying to boot from the nic. During this time, however, the VIO on the FPGA showed that the FPGA's low level Xilinx PCIe code had the link UP. So the PCIe bus wasn't totally messed up, just a little messed up. To recap, the link was up but the bus seems hung. Subsequently, I took the working Xilinx demo and commented out the "app"lication. Then that one started hanging the host boot as well. This makes me think that the PCIe bus is getting hung up due to some higher level transaction situation, at a level above "link up", and at a level that the Xilinx demo "app"lication and analogously the xillybus_core handle. Call both those "applications". So these applications must do some transactions and when they fail to do so or fail to do it right, the PCIe bus gets hung up enough, even though the link is up, to hang the remainder of the boot.

* You have said *indirectly* that the xillybus_core *should* be agnostic about number of lanes. Thanks for the info. You're surely correct, but in spite of that my testing is strongly suggesting otherwise. I will keep this conflict of info in mind.

* You've pointed to the ZC706 bundle as having a xillybus_core known tested and working on a Kintex-7 with 4 lines. Great. I'll go swap it out and see if that helps. I'll let you know the results.

* You discuss the timing of FPGA configuration versus PC booting. We are up-to-speed on that. We have reliable config-from-flash occurring after cold start. However, we've found and researched to confirm that the jtag connection must be disconnected at re-power, and we're using the working Xilinx PCIe demo in the config memory. We've all said that the FPGA must be configured in advance, and our research/experiments tell us this is because it is the BIOS that does PCIe enumeration only at cold start. Given a good cold start and PCIe enumeration, we've successfully re-configured the FPGA via jtag, and then found as predicted that merely a warm start gets it up again. That's for a working bitstream, of course.

-Helmut
Guest
 

Re: Trouble getting x4 lanes working

Postby Guest » Thu Aug 30, 2018 7:48 pm

Ah ha, I see at http://xillybus.com/pcie-download that the Zynq 7000 (PCIe) demo for the ZC706 is "available upon request".

I am respectfully requesting to get a copy of that demo. My intent is to pull that xillybus_core out and drop it into my best-so-far xillydemo project, in hopes of getting 4 lanes working.

If you need my contact info, such as phone or remote office address, please email me at hforren@mteq.com.

Thanks very much,
Helmut
Guest
 

Re: Trouble getting x4 lanes working

Postby Guest » Thu Aug 30, 2018 11:29 pm

SOLVED.

Thanks for the Zynq 7000 / ZC706 version, Eli.

We just got a Xillybus-based demo to get recognized by linux with lspci. This was from jtag programming and warm boot then lspci, having previously had a good cold start pulling the xilinx pcie demo from config memory. We've tested this successfully twice. (NOTE: the straight lanes works while the crossed lanes doesn't. So now we know the answer to that question.)

Then we wrote this Xillybus-based demo to config memory, THIS ALSO WORK!

Tomorrow, I'll subsequently try attaching a driver.

INTERESTING: This working Xillybus-based demo comes from the original Xillybus Zynq 7000 demo, where the Zynq 7000 is a Kintex-7 plus ARM. But it begins our life as x4 lanes, so I didn't have to edit the number of lanes. It built out of the box. I changed to 160T and built. I moved lanes to the correct pins. Then it worked on our custom target board.

CONFUSING HOWEVER, all the xillybus demos had a wrapper around the Xilinx endpoint. For the original KC705 demo, I edited that wrapper down to 4 lanes and reconfigured the IP for four lanes. LO AND BEHOLD, the Zynq 7000 demo that works has 8 lane wrappers around a 4 lane IP. The wrappers have 8-bit vectors for the lane variables, being passed into 4-bit vector inputs of the IP. Somehow this works.

Perhaps my theory about xillybus_core not working in our 4-lane case while working in the KC705 8-lane case is this. The xillybus_core needs the wrappers to be 8 lanes wide. When not 8 lanes wide, problems occur. Now, the xillybus_core (that's supposed to be identical for KC705 and Zyna 7000) is talking to an 8 lane wrapper and not causing trouble, then somehow the 8 lane wrapper can invoke the 4 lane IP and work. (Otherwise I made an error in reducing the wrapper, but I was very careful, didn't change much, and don't think I made an error.)

NEXT, I could analyze this all further to exactly understand, including migrating back to my failing xillybus demo to get it to work. HOWEVER, I will NOT spend that time. I'll take what works and race forward...

-Helmut
Guest
 

Re: Trouble getting x4 lanes working

Postby support » Fri Aug 31, 2018 7:45 am

Hello,

Nice to hear that the ZC706 bundle I sent you helped. Now, let's get a couple of issues rectified:

Guest wrote:CONFUSING HOWEVER, all the xillybus demos had a wrapper around the Xilinx endpoint. For the original KC705 demo, I edited that wrapper down to 4 lanes and reconfigured the IP for four lanes. LO AND BEHOLD, the Zynq 7000 demo that works has 8 lane wrappers around a 4 lane IP. The wrappers have 8-bit vectors for the lane variables, being passed into 4-bit vector inputs of the IP. Somehow this works.

Actually, the story is this: The ZC706 bundle is derived from the KC705 bundle. The support for ISE (an old Xilinx tool) was dropped, so one of the wrappers, which adapts the port names was removed from the bundle. Indeed, the vector width of the PCIe lanes of the wrappers is 8 and not 4, but the vector width presented by Xilinx' PCIe block is 4, and the vector width at the top level is 4 as well. Those extra 4 wires have no connections at either end, and are eliminated by the synthesizer.

Having these four dangling wires isn't the most elegant way, but it's harmless. Besides, this bundle is a hack for those few who use ZC706 as a PCIe board, so there wasn't much emphasize on the aesthetics.

So no, the vector width thing has nothing to do with your ups and downs. I would suggest looking at if and how your own conversion to 4x differs from one in the ZC706 bundle.

Regards,
Eli
support
 
Posts: 623
Joined: Tue Apr 24, 2012 3:46 pm

Re: Trouble getting x4 lanes working

Postby Guest » Fri Aug 31, 2018 4:44 pm

Thanks very much for the clarification, Eli.

Yes, it's indeed possible that my edit had an error. I do have LOTS of experience doing that kind of thing, but errors do still happen.

Well, it's working now and I'm behind deadline. So I'll have to just assume it was my error and move on, using the working version!

FYI, just a little bit ago I got my large built-up-and-tested KC705 project working *as well* on our custom 4-lane board. All the /dev/files show up, and I can test some bi-dir comms through them. This means that *finally* we're on the verge of being able to actually test the project in the real world on the custom board.

Thanks very much for Xillybus in general, and for your help in getting this far.
-Helmut
Guest
 


Return to Xillybus

cron