Initial Environment:
Developing on Windows 10
LiberoSOC 2021.3
Softconsole 2021.3
HSS 2022.3 : Binary downloaded from git, referenced from libero project
BSP Linux 2022.3: wic.gz downloaded from git, loaded onto SD card and eMMC
FPGA 2022.3 reference design, built from git sources.
I am capable of booting into linux when using all reference binaries. I am capable of booting into linux when I build HSS and Reference FPGA from sources.
Problem: After modifying FPGA SmartDesign, the HSS fails to proceed past the âAttempting to select SDCARDâŚâ line.
My design update seeks to accomplish the following: allow Linux application to access a set of sensors on a custom board that electrically/mechanically fits the Raspberry Pi connector. The actions I perform:
Create new MSS based on linux-icicle configuration, but with FIC1 enabled (I need a different clock frequency than those generated for FIC0 and FIC2 for the sensor interfaces). Generate the XML and the czx.
Copy new MSS XML into appropriate board directory, follow instructions to copy defconfig into main directory.
Rebuild the HSS.
Open FPGA design in LiberoSOC
Rename any RPi signals to match my sensor interface IP core ports, for design readability.
Update RPI.pdc constraints to match signal names and direction.
Import new MSS, and update instance.
Update ClocksAndResets to generate my sensor clocks and resets based on the 160MHz Oscillator output to Global Buffers. Add output ports, regenerate the SD.
Create new AXI4 Interface (1 initiator, 1 target). Connect FIC1 on MSS to it Initiator port, and IP core to target port.
Connect all pins to my IO coreâs ports.
Update all FIC1 related clocks to use the new frequencies generated.
Add 3 interrupts to the MSS interrupt vector (not changing any existing Icicle reference design interrupts).
Save and generate, regen timing constraints etc.
Build array data, update bitstream configuration to bring in HSS compiled ELF.
Program the design, and watch the HSS UART output. Stalls as described above.
I have tried builds step by step with each action, but it fails at adding the IO core- but thatâs also the first step where the new clocks, resets, interrupts and core actually get assigned (which I assume are synthesized out up to that point).
In the face of this failure, Iâve tried creating a minimalist system.
Created a new MSS based on Icicle design, except with no peripherals except ethernet and UARTs
Created a new LiberoSOC project targeting the Icicle Board.
Imported the MSS
Connected all MSS IO to pins.
Updated constraint files in project to only include the required IO to pins that actually exist in this minimalist design.
Build the HSS using the minimal UARTS/eth MSS XML
Proceed through LiberoSOC flow until I get to the point where I can add the HSS elf to the eNVM configuration.
Create and program the bitstream.
Fails in the exact same way as before.
Iâm really not sure where to take my debugging from this point.
As a further clarification: My end goal is to have only UARTs, ethernet, and my sensor IO core in the design. My application has no need for GPIO, PCIe, i2c, SPI, CAN etc. Simply Linux, running an application that records sensor data, and with Linux CLI access via UART and/or Eth (SSH), and fileIO.
I am most interested in getting a minimal build set up, so I can incrementally add to a functional minimal build.
I have made no modifications to the reference Linux system image - but this issue seems to appear before the HSS makes the jump to the payload. The HSS statement âAttempting to select the SDCARDâŚâ never yields as passed/failed response.
I think the issue youâre facing here is that the SDIO register isnât in your minimal design (Iâm not sure about the modified reference design at the moment). You should see a HDL+ core in the reference design called âSDIO_REGISTERâ, this is just a single bit on FIC3 which is used to select between the SD card and eMMC by driving the mux select signals on the board. The HSS sets / unsets it when it selects between the two memories. If its not in your minimal design the write to FIC3 will never return which will cause the HSS to hang and result in the attempt to select never returning. Could you try adding this to your minimal design (even not connecting it if you donât want to) just to see if this is in fact the issue?
Fantastic advice. I had no idea that SDIO_REGISTER was an essential component, so I most certainly did not include it in the minimal design. Iâll try it and report back. Appreciate a glimmer of hope that didnât involve me putting printfâs into every 10 lines of the HSS mmc init code!
FIC0 and FIC3 initiator interfaces are present, no FIC1,FIC2.
HSS Compile:
starting with the 2022.3 tagged source code
copy def_config to root directory as .config
copy .xml from generated MSS into boards/mpfs-icicle-kit-es/soc_fpga_design/xml
build the Default project.
Block Diagram:
Starting with clean 2022.3 Icicle build script.
removed everything related FIC0 (PCIe, LSRAM, and all FIC0 bus-related blocks).
modified CLOCKS_AND_RESETS: now only generates 2 clocks and 2 resets, derived from 160MHz GL. Clock0 is used for FIC0, and Clock1 (20 MHz, lower speed than FIC0) is used for FIC3.
Removed corei2c
Removed all pins related to PCIe, i2c, CAN, QSPI
Updated constraint files to either remove unused files, or comment out unused IO pin definitions as applicable.
Ensured that the APB high-low mux, CoreGPIO (LEDs), SDIO_REGISTER, CoreUARTapb remain.
confirmed that the E51 and U54 block was unchanged, all interfaces were connected properly, and all interrupts wired to the MSS per the reference
Reviewed clocks and resets to make sure polarity looks good, and I didnât accidentally wire clocks to resets, resets to clocks, FIC0 clock to FIC3 clock and FIC3 to FIC0.
ran âclean and re-runâ through Generate FPGA Array Data
opened âConfigure Design Initializationâ and selected workspace/hart-software-services/Default/hss-envm-wrapper.hex as the contents of the eNVM tab.
Ran âgenerate bitstreamâ
Copied the resulting ppd file to the computer which is connected to the Icicle board (my dev computer is MUCH faster than my lab computer for compilations).
Selected new programming file in FPExpress project.
Hit PROGRAM
Run into same failure at âAttempting to select SDCARD âŚâ
This is where I think its getting stuck. Even if the SD card fails to initialize you should get a return from the mmc_init_common so I donât think its getting that far.
Your steps all look good to be honest! There is a PCIe init feature in the HSS youâll have to disable as you removed PCIe but that wouldnât be causing this issue.
What you could do (as a workaround just to get your build booting) is disable the CONFIG_SERVICE_SDIO_REGISTER_PRESENT feature in the HSS (this means it wonât try and write the register on FIC3) and then also only choose either an eMMC or SD target for the HSS boot and tie the mux select signals high or low depending on whether you want eMMC or SD. This should get it past this road block as a sanity check.
Is the clock for FIC3 >= 125MHz in your design by any chance? You mentioned its 20MHz below FIC0 but Iâm not sure what thats running at.
Just to check a few more things (I have an idea now of what it might be):
Have you tied âSDIO_SW_EN_Nâ low? Its an output pin in the reference design which is used to enable the mux, without it things might fail. I donât think this is the issue, just another gotya.
Have you disabled the DLL for FIC0? The DLL is only needed when the frequency is >= 125MHz, if youâre using the DLL locks as part of your reset structure (like we do in the reference design) its probably not going to lock and thatâll hold the logic driven by any reset using the DLL lock signal in reset which could explain why the SDIO register still isnât working.
To resolve this you need to update the MSS configuration to disable the DLL and import the updated MSS component into Libero, the DLLs are configured using flash bits and not the XML so this change doesnât mean you have to re-build the HSS just re-generate the design
To resolve the IHC its similar to the SDIO side, its looking for the IHC Subsystem from the reference design, you can disable the IHC support in the HSS config to bypass using it for now if you need or add in the subsystem to your design
Would you agree that this is pointing to a fundamental issue with the FIC3 initiator? It appears to stall while accessing the IHC_Subsystem and SDIO_REGISTER peripherals. Could there be an issue with having removed a few APB peripherals but not removed their ports (marked unused instead)?
If so, Iâm not sure how/where to begin this troubleshooting effort. I looked at the LiberoSOC memory map, and the SDIO_REGISTER (for example) appeared as expected at 0x4F00_0000. That lines up with the FIC3 space (0x4000_0000) and the defined offset for SDIO_REGISTER at 0x0F00_0000.
To be honest this shouldnât be happening, did you see my post above re the DLL locks used in the reset structure? I donât think its the removed peripherals as they arenât being used / accessed from the point in software youâre at. From what youâre describing the SDIO register / IHC subsystem / fabric are in reset or just inaccessible. The reason for the lockup is that a hart (in this case the E51) is doing a write / read on the APB interface and waiting for the pready signal to assert. If the peripheral isnât at the correct address or is in reset that signal never asserts and the hart hangs waiting for it to come back. One way around this would be having a counter on the interface and if it times out just return 0 and carry on, the danger with that though is that how do you know a peripheral isnât just slow returning the transaction vs non existent / not working.
In the source for the register the only times you wonât get pready back are:
reset
no access
Otherwise on any read / write that reaches the register pready will assert to complete the transaction.
It points to an issue in something related to the design / software as it shouldnât access something thats not there or in reset. A solution for part of this is using PMPs to protect the hart from trying to access something it shouldnât and only grant it access to areas of memory it should be accessing and if it goes outside this itâll trap instead of hanging.
To check this theory you could pull the reset line for the FIC3 peripherals out to an LED and see if its on / off to check the reset state.
Sorry - I missed this post! I will revisit those suggestions.
Update: Running HSS config (make BOARD=MPFS-ICICLE-KIT-ES config), I removed SDIO_REGISTER checking, removed IHC, and removed PCIe init. This allowed me to complete HSS loading and handoff to linux kernel, where it stalled at âStarting kernel âŚâ. It does not start the âtimestampedâ messages such as calling out the Kernel revision.
On my next update, I am going to:
I had DLL enabled. I have since disabled it - I didnât realize that was only essential for f >= 125MHz. This could well be the cause of those subsystems remaining in reset.
I will output that reset to an LED. I work remote, but I have a teammate who will be able to check the status. This is a fallback plan for if the DLL lifting doesnât work.
No worries, hopefully thats it. We are updating the MSS configurator to add a tool tip to try and highlight when the DLL is needed or not to try and make it a bit more obvious when you need this option
Removing DLL from MSS seems to have worked and removed the reset issue.
I am now able to load through the HSS using the stock HSS (default Icicle .config, no modifications to any source code etc). This means that âBuild Config â General Configuration Options â Miscellaneous â Setup PCIeâ is still enabled.
This didnât prevent completion of the HSS and handoff to linux. Unfortunately Iâm not linux-y enough to know if that is causing my stall at âStarting KernelâŚâ Iâm going to remove âSetup PCIeâ from the config, rebuild HSS, and re-try to see if the kernel is hanging because of this.
Otherwise, I may need to bring in one of my teammates working the software part of the project to help me work through this if there are any modifications to the Yocto image.
Excellent! Interesting that set up PCIe succeeded but still its not needed in your case so better safe than sorry. Iâm not sure why youâre getting the hang either, youâll need to modify the fabric device tree to take out the un-necessary peripherals that were removed from the fabric as this will probably cause an issue with the boot, I wouldnât expect it at the stage youâre at but you never know!
Do you have a recommendation for whether I should use Yocto BSP or Buildroot system for generating the appropriate linux build? Iâm a competent user of linux, and have built kernels in the past for desktop systems (Gentoo, Ubuntu etc) - but this will be my first attempt at tailoring it to an embedded system.
Another question: how do I buy you a coffee/tea/pint? Thanks a ton for getting me to this point. Itâs going to take me some time to get the Linux build environment setup, so I may go dormant for a bit.
Six to one half a dozen to the other⌠Yocto seems to be the more popular choice at the moment but Buildroot is a bit easier to get started with and modify. If you just want to figure out the mods needed to get Linux booting Buildroot might be the easier option, but itâll only give you initramfs so any changes / add ons you make while Linux is running wonât be persistent.
Ha! Just doing my job, if you ever end up in Dublin let me know and we can grab a Guinness!
I am still unable to progress into the linux kernel. We get through Uboot which attempts to hand-off to linux, but error remains.
Steps:
Using the minimized design (no PCIe, LSRAM, i2c, PWM, etc) in fabric, and with i2c, SPI, QSPI, CAN disabled in MSS.
Removing the few devices from mpfs-icicle-kit-fabric.dts and recompiling didnât seem to result in any progress.
Removing the devices in mpfs-icicle-kit.dts and mpfs.dts that were removed from the MSS configuration also yielded no improvement.
Running that recompiled kernel on a stock reference Icicle Kit FPGA also failed to make any progress.
I suppose my next step is to perhaps to build something like this:
MSS is tailored only to modify the Use DLLs with my lower clock freqs, and enable FIC1
Fabric is modified to remove LSRAM, corePWM, coreI2c, PCIe and any blocks related to the PCIe such as the dynamic reconfiguration block . Remove top level pins for those interfaces (USB will remain).
Tie all unused inputs to zero, mark all outputs as âunusedâ. Update Clock and Reset block appropriately (removing all PCIe related clocks, and introducing all my intended system clocks/resets).
Modify HSS config so that Setup PCIe is removed, use SD Card is removed (eMMC remains) so that I can fully remotely update my device.
Modify only the mpfs-icicle-kit-fabric.dtsi to remove the corePWM, coreI2c2, fpgalsram.
Questions:
Would you propose any other techniques for getting past the linux-kernel-wonât-start issue Iâm having? My intuition is clearly not lined up with what it actually takes to create a stripped down PFSOC system.
Fabric Clock 1 relates to FIC0 aclk, Fabrick Clock 3 relates to FIC2 aclk? If Iâm using FIC1 as well, must I add an entry into this dtsi
Thanks! Barrel of guiness for you when we figure this out.
OldCat
Would you propose any other techniques for getting past the linux-kernel-wonât-start issue Iâm having?
The best recommended methodology would be to bank your current design and go to a stock reference design and modify the Linux image to systematically remove the peripherals you donât want one at a time.
At least this way if there is something causing an issue when removed you should see it, it could be a combination of factors preventing the boot so its very hard to see which one is actually the culprit.
It seems a bit like Linux is trying to access something that isnât there, Iâm not sure what peripheral this could be to be honest.
Then at least by the end of doing the steps above you should have a working minimal kernel thatâll boot on the design you want to use.
Fabric Clock 1 relates to FIC0 aclk, Fabrick Clock 3 relates to FIC2 aclk? If Iâm using FIC1 as well, must I add an entry into this dtsi
