Modified reference design fails to pass "Attempting to select SDCARD..."

Initial Environment:
Developing on Windows 10
LiberoSOC 2021.3
Softconsole 2021.3
HSS 2022.3 : Binary downloaded from git, referenced from libero project
BSP Linux 2022.3: wic.gz downloaded from git, loaded onto SD card and eMMC
FPGA 2022.3 reference design, built from git sources.

I am capable of booting into linux when using all reference binaries. I am capable of booting into linux when I build HSS and Reference FPGA from sources.

Problem: After modifying FPGA SmartDesign, the HSS fails to proceed past the “Attempting to select SDCARD…” line.

My design update seeks to accomplish the following: allow Linux application to access a set of sensors on a custom board that electrically/mechanically fits the Raspberry Pi connector. The actions I perform:

  1. Create new MSS based on linux-icicle configuration, but with FIC1 enabled (I need a different clock frequency than those generated for FIC0 and FIC2 for the sensor interfaces). Generate the XML and the czx.
  2. Copy new MSS XML into appropriate board directory, follow instructions to copy defconfig into main directory.
  3. Rebuild the HSS.
  4. Open FPGA design in LiberoSOC
  5. Rename any RPi signals to match my sensor interface IP core ports, for design readability.
  6. Update RPI.pdc constraints to match signal names and direction.
  7. Import new MSS, and update instance.
  8. Update ClocksAndResets to generate my sensor clocks and resets based on the 160MHz Oscillator output to Global Buffers. Add output ports, regenerate the SD.
  9. Create new AXI4 Interface (1 initiator, 1 target). Connect FIC1 on MSS to it Initiator port, and IP core to target port.
  10. Connect all pins to my IO core’s ports.
  11. Update all FIC1 related clocks to use the new frequencies generated.
  12. Add 3 interrupts to the MSS interrupt vector (not changing any existing Icicle reference design interrupts).
  13. Save and generate, regen timing constraints etc.
  14. Build array data, update bitstream configuration to bring in HSS compiled ELF.
  15. Program the design, and watch the HSS UART output. Stalls as described above.

I have tried builds step by step with each action, but it fails at adding the IO core- but that’s also the first step where the new clocks, resets, interrupts and core actually get assigned (which I assume are synthesized out up to that point).

In the face of this failure, I’ve tried creating a minimalist system.

  1. Created a new MSS based on Icicle design, except with no peripherals except ethernet and UARTs
  2. Created a new LiberoSOC project targeting the Icicle Board.
  3. Imported the MSS
  4. Connected all MSS IO to pins.
  5. Updated constraint files in project to only include the required IO to pins that actually exist in this minimalist design.
  6. Build the HSS using the minimal UARTS/eth MSS XML
  7. Proceed through LiberoSOC flow until I get to the point where I can add the HSS elf to the eNVM configuration.
  8. Create and program the bitstream.

Fails in the exact same way as before.

I’m really not sure where to take my debugging from this point.

Thanks for suggestions!

@hugh.breslin

As a further clarification: My end goal is to have only UARTs, ethernet, and my sensor IO core in the design. My application has no need for GPIO, PCIe, i2c, SPI, CAN etc. Simply Linux, running an application that records sensor data, and with Linux CLI access via UART and/or Eth (SSH), and fileIO.

I am most interested in getting a minimal build set up, so I can incrementally add to a functional minimal build.

I have made no modifications to the reference Linux system image - but this issue seems to appear before the HSS makes the jump to the payload. The HSS statement “Attempting to select the SDCARD…” never yields as passed/failed response.

Hi @OldCat :slight_smile:

I think the issue you’re facing here is that the SDIO register isn’t in your minimal design (I’m not sure about the modified reference design at the moment). You should see a HDL+ core in the reference design called “SDIO_REGISTER”, this is just a single bit on FIC3 which is used to select between the SD card and eMMC by driving the mux select signals on the board. The HSS sets / unsets it when it selects between the two memories. If its not in your minimal design the write to FIC3 will never return which will cause the HSS to hang and result in the attempt to select never returning. Could you try adding this to your minimal design (even not connecting it if you don’t want to) just to see if this is in fact the issue?

Kind regards,
Hugh

1 Like

Fantastic advice. I had no idea that SDIO_REGISTER was an essential component, so I most certainly did not include it in the minimal design. I’ll try it and report back. Appreciate a glimmer of hope that didn’t involve me putting printf’s into every 10 lines of the HSS mmc init code!

1 Like

My minimal design fails at the same point.

MSS Config:

  • all UARTs unchanged
  • ethernets unchanged
  • GPIO unchanged
  • FIC0 and FIC3 initiator interfaces are present, no FIC1,FIC2.

HSS Compile:

  • starting with the 2022.3 tagged source code
  • copy def_config to root directory as .config
  • copy .xml from generated MSS into boards/mpfs-icicle-kit-es/soc_fpga_design/xml
  • build the Default project.

Block Diagram:

  • Starting with clean 2022.3 Icicle build script.
  • removed everything related FIC0 (PCIe, LSRAM, and all FIC0 bus-related blocks).
  • modified CLOCKS_AND_RESETS: now only generates 2 clocks and 2 resets, derived from 160MHz GL. Clock0 is used for FIC0, and Clock1 (20 MHz, lower speed than FIC0) is used for FIC3.
  • Removed corei2c
  • Removed all pins related to PCIe, i2c, CAN, QSPI
  • Updated constraint files to either remove unused files, or comment out unused IO pin definitions as applicable.
  • Ensured that the APB high-low mux, CoreGPIO (LEDs), SDIO_REGISTER, CoreUARTapb remain.
  • confirmed that the E51 and U54 block was unchanged, all interfaces were connected properly, and all interrupts wired to the MSS per the reference
  • Reviewed clocks and resets to make sure polarity looks good, and I didn’t accidentally wire clocks to resets, resets to clocks, FIC0 clock to FIC3 clock and FIC3 to FIC0.
  • marked FIC0 as unused.
  • marked open outputs of FIC3 initiator as unused.
  • force all unused interrupts and GPIO pins low
  • confirmed FIC3 initiator block settings matched reference design

Build Flows:

  • Generated all smartdesign components.
  • rebuilt hierarchy
  • ‘derived’ all timing constraints
  • ran “clean and re-run” through Generate FPGA Array Data
  • opened “Configure Design Initialization” and selected workspace/hart-software-services/Default/hss-envm-wrapper.hex as the contents of the eNVM tab.
  • Ran “generate bitstream”
  • Copied the resulting ppd file to the computer which is connected to the Icicle board (my dev computer is MUCH faster than my lab computer for compilations).
  • Selected new programming file in FPExpress project.
  • Hit PROGRAM

Run into same failure at “Attempting to select SDCARD …”

Weird…

This is where I think its getting stuck. Even if the SD card fails to initialize you should get a return from the mmc_init_common so I don’t think its getting that far.

Your steps all look good to be honest! There is a PCIe init feature in the HSS you’ll have to disable as you removed PCIe but that wouldn’t be causing this issue.

What you could do (as a workaround just to get your build booting) is disable the CONFIG_SERVICE_SDIO_REGISTER_PRESENT feature in the HSS (this means it won’t try and write the register on FIC3) and then also only choose either an eMMC or SD target for the HSS boot and tie the mux select signals high or low depending on whether you want eMMC or SD. This should get it past this road block as a sanity check.

Is the clock for FIC3 >= 125MHz in your design by any chance? You mentioned its 20MHz below FIC0 but I’m not sure what thats running at.

I will attempt the following and report back.

  • tie SDIO_SW_SEL0,1 high in block diagram
  • remove the CONFIG_SERVICE_SDIO_REGISTER_PRESENT feature in HSS
  • remove the PCIe Init feature in HSS.

For clarity, my FIC0 frequency is 75 MHz, and my FIC3 frequency is 20 MHz.

Just to check a few more things (I have an idea now of what it might be):

  1. Have you tied “SDIO_SW_EN_N” low? Its an output pin in the reference design which is used to enable the mux, without it things might fail. I don’t think this is the issue, just another gotya.

  2. Have you disabled the DLL for FIC0? The DLL is only needed when the frequency is >= 125MHz, if you’re using the DLL locks as part of your reset structure (like we do in the reference design) its probably not going to lock and that’ll hold the logic driven by any reset using the DLL lock signal in reset which could explain why the SDIO register still isn’t working.
    To resolve this you need to update the MSS configuration to disable the DLL and import the updated MSS component into Libero, the DLLs are configured using flash bits and not the XML so this change doesn’t mean you have to re-build the HSS just re-generate the design

Latest attempt:

  • tied SDIO_SW_SEL0,1 high in block diagram.
  • mmc_api.c : commented out the line in the ifdef.
  • no change to PCIe init - just doing one thing at a time.

IT PASSES THAT POINT IN THE HSS!

Presently stalled following HSS_IHCInit(): Initializing Mi-V IHC.

Next step: removing PCIe init.

1 Like

Woohoo! :partying_face:

To resolve the IHC its similar to the SDIO side, its looking for the IHC Subsystem from the reference design, you can disable the IHC support in the HSS config to bypass using it for now if you need or add in the subsystem to your design

Would you agree that this is pointing to a fundamental issue with the FIC3 initiator? It appears to stall while accessing the IHC_Subsystem and SDIO_REGISTER peripherals. Could there be an issue with having removed a few APB peripherals but not removed their ports (marked unused instead)?

If so, I’m not sure how/where to begin this troubleshooting effort. I looked at the LiberoSOC memory map, and the SDIO_REGISTER (for example) appeared as expected at 0x4F00_0000. That lines up with the FIC3 space (0x4000_0000) and the defined offset for SDIO_REGISTER at 0x0F00_0000.

Its a feature not a bug :wink:

To be honest this shouldn’t be happening, did you see my post above re the DLL locks used in the reset structure? I don’t think its the removed peripherals as they aren’t being used / accessed from the point in software you’re at. From what you’re describing the SDIO register / IHC subsystem / fabric are in reset or just inaccessible. The reason for the lockup is that a hart (in this case the E51) is doing a write / read on the APB interface and waiting for the pready signal to assert. If the peripheral isn’t at the correct address or is in reset that signal never asserts and the hart hangs waiting for it to come back. One way around this would be having a counter on the interface and if it times out just return 0 and carry on, the danger with that though is that how do you know a peripheral isn’t just slow returning the transaction vs non existent / not working.

In the source for the register the only times you won’t get pready back are:

  1. reset
  2. no access

Otherwise on any read / write that reaches the register pready will assert to complete the transaction.

It points to an issue in something related to the design / software as it shouldn’t access something thats not there or in reset. A solution for part of this is using PMPs to protect the hart from trying to access something it shouldn’t and only grant it access to areas of memory it should be accessing and if it goes outside this it’ll trap instead of hanging.

To check this theory you could pull the reset line for the FIC3 peripherals out to an LED and see if its on / off to check the reset state.

Sorry - I missed this post! I will revisit those suggestions.

Update: Running HSS config (make BOARD=MPFS-ICICLE-KIT-ES config), I removed SDIO_REGISTER checking, removed IHC, and removed PCIe init. This allowed me to complete HSS loading and handoff to linux kernel, where it stalled at “Starting kernel …”. It does not start the ‘timestamped’ messages such as calling out the Kernel revision.

On my next update, I am going to:

  1. I had DLL enabled. I have since disabled it - I didn’t realize that was only essential for f >= 125MHz. This could well be the cause of those subsystems remaining in reset.

  2. I will output that reset to an LED. I work remote, but I have a teammate who will be able to check the status. This is a fallback plan for if the DLL lifting doesn’t work.

  3. Revert the original .config for the Icicle Board.

No worries, hopefully thats it. We are updating the MSS configurator to add a tool tip to try and highlight when the DLL is needed or not to try and make it a bit more obvious when you need this option :slight_smile:

Let me know how it goes!

Removing DLL from MSS seems to have worked and removed the reset issue.

I am now able to load through the HSS using the stock HSS (default Icicle .config, no modifications to any source code etc). This means that “Build Config → General Configuration Options → Miscellaneous → Setup PCIe” is still enabled.

This didn’t prevent completion of the HSS and handoff to linux. Unfortunately I’m not linux-y enough to know if that is causing my stall at “Starting Kernel…” I’m going to remove “Setup PCIe” from the config, rebuild HSS, and re-try to see if the kernel is hanging because of this.

Otherwise, I may need to bring in one of my teammates working the software part of the project to help me work through this if there are any modifications to the Yocto image.

1 Like

Excellent! Interesting that set up PCIe succeeded but still its not needed in your case so better safe than sorry. I’m not sure why you’re getting the hang either, you’ll need to modify the fabric device tree to take out the un-necessary peripherals that were removed from the fabric as this will probably cause an issue with the boot, I wouldn’t expect it at the stage you’re at but you never know!

Do you have a recommendation for whether I should use Yocto BSP or Buildroot system for generating the appropriate linux build? I’m a competent user of linux, and have built kernels in the past for desktop systems (Gentoo, Ubuntu etc) - but this will be my first attempt at tailoring it to an embedded system.

Another question: how do I buy you a coffee/tea/pint? Thanks a ton for getting me to this point. It’s going to take me some time to get the Linux build environment setup, so I may go dormant for a bit.

Six to one half a dozen to the other… Yocto seems to be the more popular choice at the moment but Buildroot is a bit easier to get started with and modify. If you just want to figure out the mods needed to get Linux booting Buildroot might be the easier option, but it’ll only give you initramfs so any changes / add ons you make while Linux is running won’t be persistent.

Ha! Just doing my job, if you ever end up in Dublin let me know and we can grab a Guinness!

@hugh.breslin ,

I am still unable to progress into the linux kernel. We get through Uboot which attempts to hand-off to linux, but error remains.

Steps:

  1. Using the minimized design (no PCIe, LSRAM, i2c, PWM, etc) in fabric, and with i2c, SPI, QSPI, CAN disabled in MSS.
  2. Removing the few devices from mpfs-icicle-kit-fabric.dts and recompiling didn’t seem to result in any progress.
  3. Removing the devices in mpfs-icicle-kit.dts and mpfs.dts that were removed from the MSS configuration also yielded no improvement.
  4. Running that recompiled kernel on a stock reference Icicle Kit FPGA also failed to make any progress.

I suppose my next step is to perhaps to build something like this:

  • MSS is tailored only to modify the Use DLLs with my lower clock freqs, and enable FIC1
  • Fabric is modified to remove LSRAM, corePWM, coreI2c, PCIe and any blocks related to the PCIe such as the dynamic reconfiguration block . Remove top level pins for those interfaces (USB will remain).
  • Tie all unused inputs to zero, mark all outputs as ‘unused’. Update Clock and Reset block appropriately (removing all PCIe related clocks, and introducing all my intended system clocks/resets).
  • Modify HSS config so that Setup PCIe is removed, use SD Card is removed (eMMC remains) so that I can fully remotely update my device.
  • Modify only the mpfs-icicle-kit-fabric.dtsi to remove the corePWM, coreI2c2, fpgalsram.

Questions:

  • Would you propose any other techniques for getting past the linux-kernel-won’t-start issue I’m having? My intuition is clearly not lined up with what it actually takes to create a stripped down PFSOC system.
  • Fabric Clock 1 relates to FIC0 aclk, Fabrick Clock 3 relates to FIC2 aclk? If I’m using FIC1 as well, must I add an entry into this dtsi

Thanks! Barrel of guiness for you when we figure this out.
OldCat

Hi again @OldCat :slight_smile:

Sounds like fun!

Would you propose any other techniques for getting past the linux-kernel-won’t-start issue I’m having?

The best recommended methodology would be to bank your current design and go to a stock reference design and modify the Linux image to systematically remove the peripherals you don’t want one at a time.

At least this way if there is something causing an issue when removed you should see it, it could be a combination of factors preventing the boot so its very hard to see which one is actually the culprit.

It seems a bit like Linux is trying to access something that isn’t there, I’m not sure what peripheral this could be to be honest.

Then at least by the end of doing the steps above you should have a working minimal kernel that’ll boot on the design you want to use.

Fabric Clock 1 relates to FIC0 aclk, Fabrick Clock 3 relates to FIC2 aclk? If I’m using FIC1 as well, must I add an entry into this dtsi

Yes the DTSI should be aware of each source clock