Booting Linux and bare-metal at the same time, by hss-payload-generator

Hello,

Please allow my long post below.

I’m trying hss-payload-generator in HSS (release 2020.10) to boot U-Boot and also bare-metal program, on different harts, at the same time, but it doesn’t work…

I’m using polarfire-soc-buildroot-sdk (commit 6514a11) to generate U-Boot and Linux kernel for SD-card.

I modified icicle-kit-es-sd.dts as following so that Linux only uses U54_1 and U54_2.

diff --git a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
index 8443d55..9052179 100644
--- a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
+++ b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
@@ -92,7 +92,7 @@
                         reg = <3>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L17: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
@@ -117,7 +117,7 @@
                         reg = <4>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L21: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";

The following hss-payload-generator config works fine. (U-Boot and Linux boots successfully.)

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x80200000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  polarfire-soc-buildroot-sdk/work/u-boot-smode/u-boot.bin: {
    exec-addr: '0x80200000',
    owner-hart: u54_1,
    secondary-hart: u54_2,
    priv-mode: prv_s
  }

And the following also works. (Bare metal only.)

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x80200000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  my_baremetal.elf: {
    exec-addr: '0x08080000',
    owner-hart: u54_3,
    secondary-hart: u54_4,
    priv-mode: prv_m
  }

However, when combined the above as following, Linux failed to boot. (Bare metal code looks running.)

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x80200000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  my_baremetal.elf: {
    exec-addr: '0x08080000',
    owner-hart: u54_3,
    secondary-hart: u54_4,
    priv-mode: prv_m
  }
  polarfire-soc-buildroot-sdk/work/u-boot-smode/u-boot.bin: {
    exec-addr: '0x80200000',
    owner-hart: u54_1,
    secondary-hart: u54_2,
    priv-mode: prv_s
  }

Linux kernel crashes like this.

(snip)
[    0.708397] libphy: Fixed MDIO Bus: probed
[    0.714374] libphy: MACB_mii_bus: probed
[  359.359529] rcu: INFO: rcu_sched self-detected stall on CPU
[  359.365125] rcu:     0-...!: (2 ticks this GP) idle=4d2/0/0x1 softirq=28/29 fqs=0
[  359.372442]  (t=35864 jiffies g=-1119 q=2)
[  359.376557] rcu: rcu_sched kthread starved for 35864 jiffies! g-1119 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=1
[  359.386904] rcu: RCU grace-period kthread stack dump:
[  359.391960] rcu_sched       I    0    10      2 0x00000000
[  359.397459] Call Trace:
[  359.399938] [<ffffffe00040df20>] __schedule+0x218/0x504
[  359.405176] [<ffffffe00040e24e>] schedule+0x42/0xb2
[  359.410072] [<ffffffe000411050>] schedule_timeout+0x16e/0x242
[  359.415842] [<ffffffe00007aa38>] rcu_gp_kthread+0x59a/0x970
[  359.421436] [<ffffffe00004ab3a>] kthread+0xc4/0xe4
[  359.426250] [<ffffffe00002e52e>] ret_from_exception+0x0/0xc
[  359.431885] Task dump for CPU 0:
[  359.435124] swapper/0       R  running task        0     0      0 0x00000000
[  359.442187] Call Trace:
[  359.444650] [<ffffffe00002f680>] walk_stackframe+0x0/0xaa
[  359.450065] [<ffffffe00002f86c>] show_stack+0x2a/0x34
[  359.455143] [<ffffffe00004ef28>] sched_show_task+0x116/0x13c
[  359.460818] [<ffffffe00005348a>] dump_cpu_task+0x3e/0x48
[  359.466153] [<ffffffe00007c120>] rcu_dump_cpu_stacks+0x7c/0xb4
[  359.471999] [<ffffffe00007b594>] rcu_sched_clock_irq+0x40c/0x5c0
[  359.478024] [<ffffffe000080d1a>] update_process_times+0x1e/0x42
[  359.483965] [<ffffffe00008b8a8>] tick_sched_handle.isra.0+0x2a/0x3a
[  359.490246] [<ffffffe00008bb5c>] tick_sched_timer+0x4e/0x92
[  359.495831] [<ffffffe00008141c>] __hrtimer_run_queues+0xae/0x108
[  359.501853] [<ffffffe000081b16>] hrtimer_interrupt+0xca/0x1d4
[  359.507626] [<ffffffe000300532>] riscv_timer_interrupt+0x32/0x3a
[  359.513646] [<ffffffe000411e8c>] do_IRQ+0xa4/0xb8
[  359.518364] [<ffffffe00002e52e>] ret_from_exception+0x0/0xc
[  359.523944] Task dump for CPU 1:
[  359.527174] modprobe        R  running task        0    35      7 0x00000000
[  359.534228] Call Trace:
[  359.536689] [<ffffffe00002e5f4>] ret_from_kernel_thread+0x0/0x10

Connected OpenOCD and gdb, checked where harts are executing.

(gdb) info thr
  Id   Target Id                                                   Frame
* 1    Thread 1 (Name: mpfs.hart0_e51, state: target-not-halted)   0x000000002022037a in ?? ()
  2    Thread 2 (Name: mpfs.hart1_u54_1, state: target-not-halted) 0xffffffe00002e6a8 in ?? ()
  3    Thread 3 (Name: mpfs.hart2_u54_2, state: target-not-halted) 0x0000000020234e88 in ?? ()
  4    Thread 4 (Name: mpfs.hart3_u54_3, state: target-not-halted) 0x0000000008080000 in ?? ()
  5    Thread 5 (Name: mpfs.hart4_u54_4, state: target-not-halted) 0x0000000008080000 in ?? ()

By the way, source code (disassemble) of my_baremetal.elf is like this. (Simply looping at the entry point.)

$ riscv64-unknown-elf-objdump -d my_baremetal.elf

my_baremetal.elf:     file format elf64-littleriscv


Disassembly of section .text:

0000000008080000 <__text_start>:
 8080000:       0000006f                j       8080000 <__text_start>
        ...

Could you let me know where I should investigate?

Regards,
Atsushi Yokoyama

Hmmm. Your yaml all looks good, so I wonder if your second Hart did not join your Linux SMP context?

Looks like thread 1 is running the HSS< and thread 2 is in VM (Linux), and threads 3 and 4 are in your bare metal.

Where is thread 3? (U54_2) – is that in an OpenSBI call ?

Are you able to associate these addresses with symbols?

With GDB, it should be possible to use add-symbol-file and add the hss.elf / u-boot / vmlinux images to find out where exactly you are…

Also, out of curiousity, I wonder if single-Hart Linux would work okay? I’m not sure what that would tell us, other than the problem is in getting the second hart to join the first.

Hello @ivan.griffin,

Thank you for your prompt reply.

First, this time,

  • I changed .dts for U-Boot too. (Did not this yesterday.)
  • Changed number of harts for Linux to just one as you asked.
  • Regenerated .dtb for Linux, .dtb for U-Boot and also fitImage.fit.
  • Copied the fitImage.fit to /dev/sdX2 ext4 partition.

Finally, the modification to the original polarfire-soc-buildroot-sdk (commit 6514a11) is as follows.

diff --git a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
index 8443d55..9e8741e 100644
--- a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
+++ b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
@@ -67,7 +67,7 @@
                         reg = <2>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L13: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
@@ -92,7 +92,7 @@
                         reg = <3>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L17: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
@@ -117,7 +117,7 @@
                         reg = <4>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L21: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
diff --git a/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch b/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
index afef581..7f91d1c 100644
--- a/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
+++ b/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
@@ -108,7 +108,7 @@ index 0000000000..29af1f82ba
 +                      reg = <2>;
 +                      riscv,isa = "rv64imafdc";
 +                      sifive,itim = <&L14>;
-+                      status = "okay";
++                      status = "disabled";
 +                      tlb-split;
 +                      L13: interrupt-controller {
 +                              #interrupt-cells = <1>;
@@ -135,7 +135,7 @@ index 0000000000..29af1f82ba
 +                      reg = <3>;
 +                      riscv,isa = "rv64imafdc";
 +                      sifive,itim = <&L18>;
-+                      status = "okay";
++                      status = "disabled";
 +                      tlb-split;
 +                      L17: interrupt-controller {
 +                              #interrupt-cells = <1>;
@@ -162,7 +162,7 @@ index 0000000000..29af1f82ba
 +                      reg = <4>;
 +                      riscv,isa = "rv64imafdc";
 +                      sifive,itim = <&L22>;
-+                      status = "okay";
++                      status = "disabled";
 +                      tlb-split;
 +                      L21: interrupt-controller {
 +                              #interrupt-cells = <1>;

And tried the following hss-payload-generator config. (HSS is release 2020.10. Processed XML/ICICLE_MSS_SD_cfg.xml in the icicle-kit-reference-design 2020.10, by polarfire-soc-configuration-generator and copied the generated files to the HSS.)

set-name: 'PolarFire-SoC-HSS::1118-0937'

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x08080000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  my_baremetal.elf: {
    exec-addr: '0x08080000',
    owner-hart: u54_2,
    secondary-hart: u54_3,
    secondary-hart: u54_4,
    priv-mode: prv_m
  }
  polarfire-soc-buildroot-sdk/work/u-boot-smode/u-boot.bin: {
    exec-addr: '0x80200000',
    owner-hart: u54_1,
    priv-mode: prv_s
  }

Checked the output payload too.

$ ./hss-payload-generator -d payload0937.bin 
Hart Software Service formatted boot image generator v0.99.12
Copyright (c) 2020 Microchip Corporation.

opening >>payload0937.bin<<
magic:              0xb007c0de
headerLength:       0x760
chunkTableOffset:   0x610
ziChunkTableOffset: 0x700
name[0]:            >>u-boot.bin<<
entryPoint[0]:      0x80200000
privMode[0]:        1
firstChunk[0]       4
lastChunk[0]        4
numChunks[0]        1
name[1]:            >>mpfs-blinky.elf<<
entryPoint[1]:      0x8080000
privMode[1]:        3
firstChunk[1]       0
lastChunk[1]        3
numChunks[1]        4
name[2]:            >><<
entryPoint[2]:      0x8080000
privMode[2]:        3
firstChunk[2]       0
lastChunk[2]        0
numChunks[2]        0
name[3]:            >><<
entryPoint[3]:      0x8080000
privMode[3]:        3
firstChunk[3]       0
lastChunk[3]        0
numChunks[3]        0
set_name            >>PolarFire-SoC-HSS::1118-0937<<
bootImageLength:    436200
headerCrc:          0x9d2057ec
Boot Chunks: total of 2 chunks found
ZI Chunks: total of 1 chunk found

First, I’ll show the GDB session below. Please let me show an almost full copy to avoid misunderstanding.

$ riscv64-unknown-elf-gdb
GNU gdb (xPack GNU RISC-V Embedded GCC (Microsemi SoftConsole build), 64-bit) 8.3

(snip)

(gdb) target remote :3333
Remote debugging using :3333
warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x00000000202204ee in ?? ()
(gdb) set $target_riscv=1
(gdb) set mem inaccessible-by-default off
(gdb) set architecture riscv:rv64
The target architecture is assumed to be riscv:rv64
(gdb) file Default/hss.elf
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from Default/hss.elf...

(gdb) info thr
  Id   Target Id                                               Frame
* 1    Thread 1 (Name: mpfs.hart0_e51, state: debug-request)   0x00000000202204ee in RunStateMachine (pCurrentMachine=<optimized out>)
    at hss_state_machine.c:118
  2    Thread 2 (Name: mpfs.hart1_u54_1, state: debug-request) 0x0000000020234bb8 in spin_trylock (lock=0x8004278 <scratches+4344>)
    at lib/sbi/riscv_locks.c:24
  3    Thread 3 (Name: mpfs.hart2_u54_2, state: debug-request) 0x0000000008080000 in ?? ()
  4    Thread 4 (Name: mpfs.hart3_u54_3, state: debug-request) 0x0000000008080000 in ?? ()
  5    Thread 5 (Name: mpfs.hart4_u54_4, state: debug-request) 0x0000000008080000 in ?? ()

(gdb) thr 1
[Switching to thread 1 (Thread 1)]
#0  0x00000000202204ee in RunStateMachine (pCurrentMachine=<optimized out>)
    at hss_state_machine.c:118
118                         mHSS_DEBUG_PRINTF(LOG_STATE_TRANSITION, "%s::%s -> %s::%s" CRLF, pMachineName,
(gdb) where
#0  0x00000000202204ee in RunStateMachine (pCurrentMachine=<optimized out>)
    at hss_state_machine.c:118
#1  0x000000002022053c in RunStateMachines (
    spanOfPStateMachines=spanOfPStateMachines@entry=9,
    pStateMachines=pStateMachines@entry=0x20236168 <pGlobalStateMachines>)
    at hss_state_machine.c:165
#2  0x0000000020227a7e in hss_main () at hss_main.c:63
#3  0x0000000020227aa8 in main (argc=<optimized out>, argv=<optimized out>)
    at hss_main.c:81
(gdb) l
113                 {
114                     const char *pCurrentStateName =
115                         (pCurrentMachine->pStateDescs[currentState]).pStateName;
116
117                     if (prevState != currentState) {
118                         mHSS_DEBUG_PRINTF(LOG_STATE_TRANSITION, "%s::%s -> %s::%s" CRLF, pMachineName,
119                             pLastStateName, pMachineName, pCurrentStateName);
120                     }
121                 }
122             }

(gdb) thr 2
[Switching to thread 2 (Thread 2)]
#0  0x0000000020234bb8 in spin_trylock (lock=0x8004278 <scratches+4344>)
    at lib/sbi/riscv_locks.c:24
24              __asm__ __volatile__(
(gdb) where
#0  0x0000000020234bb8 in spin_trylock (lock=0x8004278 <scratches+4344>)
    at lib/sbi/riscv_locks.c:24
#1  spin_lock (lock=lock@entry=0x8004278 <scratches+4344>)
    at lib/sbi/riscv_locks.c:43
#2  0x0000000020234e84 in sbi_fifo_dequeue (
    fifo=fifo@entry=0x8004270 <scratches+4336>, 
    data=data@entry=0x8003df0 <scratches+3184>) at lib/sbi/sbi_fifo.c:174
#3  0x0000000020232320 in sbi_tlb_process_count (
    scratch=scratch@entry=0x8004200 <scratches+4224>, count=1)
    at lib/sbi/sbi_tlb.c:225
#4  0x000000002023244c in sbi_tlb_sync (scratch=0x8004200 <scratches+4224>)
    at lib/sbi/sbi_tlb.c:249
#5  0x0000000020230f32 in sbi_ipi_send (data=0x8003ef0 <scratches+3440>, 
    event=3, remote_hartid=2, scratch=0x8004200 <scratches+4224>)
    at lib/sbi/sbi_ipi.c:63
#6  sbi_ipi_send_many (scratch=scratch@entry=0x8004200 <scratches+4224>, 
    hmask=<optimized out>, hbase=hbase@entry=0, event=<optimized out>, 
    data=data@entry=0x8003ef0 <scratches+3440>) at lib/sbi/sbi_ipi.c:101
#7  0x00000000202324f6 in sbi_tlb_request (
    scratch=scratch@entry=0x8004200 <scratches+4224>, hmask=<optimized out>, 
    hbase=hbase@entry=0, tinfo=tinfo@entry=0x8003ef0 <scratches+3440>)
    at lib/sbi/sbi_tlb.c:391
#8  0x000000002022f128 in sbi_ecall_legacy_handler (
    scratch=0x8004200 <scratches+4224>, extid=<optimized out>, 
    funcid=<optimized out>, args=0x8003fc8 <scratches+3656>, 
    out_val=<optimized out>, out_trap=0x8003fa0 <scratches+3616>)
    at lib/sbi/sbi_ecall_legacy.c:103
#9  0x000000002022eda0 in sbi_ecall_handler (hartid=hartid@entry=1, 
    mcause=mcause@entry=9, regs=regs@entry=0x80040e8 <scratches+3944>, 
    scratch=scratch@entry=0x8004200 <scratches+4224>)
    at lib/sbi/sbi_ecall.c:95
#10 0x0000000020233928 in sbi_trap_handler (regs=0x80040e8 <scratches+3944>, 
    scratch=0x8004200 <scratches+4224>) at lib/sbi/sbi_trap.c:266
#11 0x0000000020220194 in _trap_handler_all_mode () at crt.S:256
Backtrace stopped: frame did not save the PC
(gdb) l
19      int spin_trylock(spinlock_t *lock)
20      {
21              const int hartid = sbi_current_hartid();
22              int tmp = hartid, busy;
23
24              __asm__ __volatile__(
25                      "       amoswap.w %0, %2, %1\n" RISCV_ACQUIRE_BARRIER
26                      : "=r"(busy), "+A"(lock->lock)
27                      : "r"(tmp)
28                      : "memory");

(gdb) thr 3   
[Switching to thread 3 (Thread 3)]
#0  0x0000000008080000 in ?? ()
(gdb) disassemble $pc, +16
Dump of assembler code from 0x8080000 to 0x8080010:
=> 0x0000000008080000:  j       0x8080000
   0x0000000008080004:  unimp
   0x0000000008080006:  unimp
   0x0000000008080008:  unimp
   0x000000000808000a:  unimp
   0x000000000808000c:  unimp
   0x000000000808000e:  unimp
End of assembler dump.

(gdb) thr 4
[Switching to thread 4 (Thread 4)]
#0  0x0000000008080000 in ?? ()
(gdb) disassemble $pc, +16
Dump of assembler code from 0x8080000 to 0x8080010:
=> 0x0000000008080000:  j       0x8080000
   0x0000000008080004:  unimp
   0x0000000008080006:  unimp
   0x0000000008080008:  unimp
   0x000000000808000a:  unimp
   0x000000000808000c:  unimp
   0x000000000808000e:  unimp
End of assembler dump.

(gdb) thr 5
[Switching to thread 5 (Thread 5)]
#0  0x0000000008080000 in ?? ()
(gdb) disassemble $pc, +16
Dump of assembler code from 0x8080000 to 0x8080010:
=> 0x0000000008080000:  j       0x8080000
   0x0000000008080004:  unimp
   0x0000000008080006:  unimp
   0x0000000008080008:  unimp
   0x000000000808000a:  unimp
   0x000000000808000c:  unimp
   0x000000000808000e:  unimp
End of assembler dump.
(gdb) 

As shown above, E51 and U54_1 look running in HSS context and OpenSBI context, respectively.

Finally, I’ll show UART output for your reference below.

  • UART 0
[1.876473] HSS_E51_Banner(): PolarFire(R) SoC Hart Software Services (HSS) - version 0.99.12
(c) Copyright 2017-2020 Microchip Corporation.

[1.892008] HSS_E51_Banner(): incorporating OpenSBI - version 0.6
(c) Copyright 2019-2020 Western Digital Corporation.

[1.904681] HSS_PrintBuildId(): Build ID: 75c200c49d32a9042c469d38ee97102f2bbb091f
[1.913795] HSS_PrintToolVersions(): Built with the following tools: 
 - riscv64-unknown-elf-gcc (xPack GNU RISC-V Embedded GCC (Microsemi SoftConsole build), 64-bit) 8.3.0
 - GNU ld (xPack GNU RISC-V Embedded GCC (Microsemi SoftConsole build), 64-bit) 2.32

[1.938707] HSS_MemTestDDRFast(): DDR size is 1 GiB
Press a key to enter CLI, ESC to skip
Timeout in 5 seconds
.....
[7.56357] HSS_TinyCLI_Parser(): CLI check timeout
[7.62433] IPI_QueuesInit(): Initializing IPI Queues (9000 bytes @ 8000e40)...
[7.70939] HSS_PMP_Init(): Initializing PMPs
[7.76494] HSS_BootInit(): Initializing Boot Image..
[7.82744] getBootImageFromMMC_(): Preparing to copy from MMC to DDR ...
[7.90729] getBootImageFromMMC_(): Attempting to read image header (1552 bytes) ...
[7.100156] GPT_ValidateHeader(): Validated GPT Header ...
[7.130536] GPT_ValidatePartitionEntries(): Validated GPT Partition Entries ...
[7.140205] copyBootImageToDDR_(): Copying 436200 bytes to 0xB0000000
[10.787715] copyBootImageToDDR_(): Calculated CRC32 of image in DDR is 16f15963
[10.877484] HSS_BootInit():  boot image passed CRC
[10.883646] HSS_BootInit(): Boot image set name: "PolarFire-SoC-HSS::1118-0937"
[10.892326] HSS_BootInit(): Boot Image registered...
[10.898663] HSS_Boot_RestartCore(): called for all harts
[10.905346] RunStateMachine(): boot_service(u54_1)::Init -> boot_service(u54_1)::SetupPMP
[10.915155] RunStateMachine(): boot_service(u54_2)::Init -> boot_service(u54_2)::SetupPMP
[10.924963] RunStateMachine(): boot_service(u54_3)::Init -> boot_service(u54_3)::SetupPMP
[10.934772] RunStateMachine(): boot_service(u54_4)::Init -> boot_service(u54_4)::SetupPMP
[10.944580] RunStateMachine(): usbdmsc_service::init -> usbdmsc_service::idle
[10.953347] RunStateMachine(): boot_service(u54_1)::SetupPMP -> boot_service(u54_1)::SetupPMPComplete
[10.964197] RunStateMachine(): boot_service(u54_2)::SetupPMP -> boot_service(u54_2)::SetupPMPComplete
[10.975047] RunStateMachine(): boot_service(u54_3)::SetupPMP -> boot_service(u54_3)::SetupPMPComplete
[10.985897] RunStateMachine(): boot_service(u54_4)::SetupPMP -> boot_service(u54_4)::SetupPMPComplete
[10.996747] RunStateMachine(): boot_service(u54_1)::SetupPMPComplete -> boot_service(u54_1)::ZeroInit
[11.07597] RunStateMachine(): boot_service(u54_2)::SetupPMPComplete -> boot_service(u54_2)::ZeroInit
[11.18360] RunStateMachine(): boot_service(u54_3)::SetupPMPComplete -> boot_service(u54_3)::ZeroInit
[11.29123] RunStateMachine(): boot_service(u54_4)::SetupPMPComplete -> boot_service(u54_4)::ZeroInit
[11.39886] RunStateMachine(): boot_service(u54_1)::ZeroInit -> boot_service(u54_1)::Download
[11.49955] RunStateMachine(): boot_service(u54_2)::ZeroInit -> boot_service(u54_2)::Download
[11.60024] RunStateMachine(): boot_service(u54_3)::ZeroInit -> boot_service(u54_3)::Download
[11.70093] RunStateMachine(): boot_service(u54_4)::ZeroInit -> boot_service(u54_4)::Download
[11.80162] RunStateMachine(): boot_service(u54_3)::Download -> boot_service(u54_3)::Idle
[11.89883] RunStateMachine(): boot_service(u54_4)::Download -> boot_service(u54_4)::Idle
[11.99605] RunStateMachine(): boot_service(u54_2)::Download -> boot_service(u54_2)::Wait
[11.109326] boot_download_chunks_onExit(): boot_service(u54_2)::u54_3:sbi_init 8080000
[11.118615] boot_download_chunks_onExit(): boot_service(u54_2)::u54_4:sbi_init 8080000
[11.127903] boot_download_chunks_onExit(): boot_service(u54_2)::u54_2:sbi_init 8080000
[11.137191] boot_wait_onEntry(): boot_service(u54_2)::Checking for IPI ACKs: - -
[11.145957] boot_wait_handler(): boot_service(u54_2)::Checking for IPI ACKs: ACK/IDLE ACK
[11.155504] RunStateMachine(): boot_service(u54_2)::Wait -> boot_service(u54_2)::Idle
[11.192064] RunStateMachine(): boot_service(u54_1)::Download -> boot_service(u54_1)::Wait
[11.201873] boot_download_chunks_onExit(): boot_service(u54_1)::u54_1:sbi_init 80200000
[11.211249] boot_wait_onEntry(): boot_service(u54_1)::Checking for IPI ACKs: - -
[11.220014] boot_wait_handler(): boot_service(u54_1)::Checking for IPI ACKs: ACK/IDLE ACK
[11.229562] RunStateMachine(): boot_service(u54_1)::Wait -> boot_service(u54_1)::Idle
(stopped here)
  • UART 1
[10.953206] HSS_Boot_PMPSetupHandler(): Hart1 setup complete
HSS_OpenSBI_Setup(): MTVEC switching from 20220230 to 20220100


U-Boot 2020.01 (Nov 18 2020 - 09:34:23 +0900)

DRAM:  1 GiB
MMC:   sdhc@20008000: 0
In:    serial@20100000
Out:   serial@20100000
Err:   serial@20100000
Net:
Warning: ethernet@20112000 using MAC address from ROM
eth0: ethernet@20112000
Hit any key to stop autoboot:  0
180 bytes read in 9 ms (19.5 KiB/s)
Running boot cmd...
16976804 bytes read in 1447 ms (11.2 MiB/s)
## Loading kernel from FIT Image at 90000000 ...
   Using 'conf@microchip_icicle-kit-es-a000-microchip.dtb' configuration
   Trying 'kernel@1' kernel subimage
     Description:  Linux kernel
     Type:         Kernel Image
     Compression:  uncompressed
     Data Start:   0x900000fc
     Data Size:    6477712 Bytes = 6.2 MiB
     Architecture: RISC-V
     OS:           Linux
     Load Address: 0x80200000
     Entry Point:  0x80200000
     Hash algo:    sha256
     Hash value:   39a5b65925451854583b9f401c47a5df9444f9efb8269bffbf4aca2b5beae2d8
   Verifying Hash Integrity ... sha256+ OK
   
(snip)

[    0.684410] Rounding down aligned max_sectors from 4294967295 to 4294967288
[    0.691881] db_root: cannot open: /etc/target
[    0.697436] libphy: Fixed MDIO Bus: probed
[    0.703246] libphy: MACB_mii_bus: probed
(stopped here)
  • UART 2
[10.964045] HSS_Boot_PMPSetupHandler(): Hart2 setup complete
HSS_OpenSBI_Setup(): MTVEC switching from 20220230 to 20220100
(stopped here)
  • UART 3
[10.974891] HSS_Boot_PMPSetupHandler(): Hart3 setup complete
HSS_OpenSBI_Setup(): MTVEC switching from 20220230 to 20220100

OpenSBI v0.6
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name          : Microchip PolarFire SoC
Platform HART Features : RV64ACDFIMSU
Platform Max HARTs     : 5
Current Hart           : 3
Firmware Base          : 0x20220000
Firmware Size          : 105 KB
Runtime SBI Version    : 0.2

MIDELEG : 0x0000000000000222
MEDELEG : 0x000000000000b109
PMP0    : 0x0000000020220000-0x000000002023ffff (A)
PMP1    : 0x0000000000000000-0x0000007fffffffff (A,R,W,X)
(stopped here)

Please let me know if you noticed some mistake of mine.

Regards,
Atsushi

May I ask if there is some update?

Regards,
Atsushi

Hi,

I went through your flow, and i don’t think you are doing anything incorrect.

When you modified the number of harts for Linux to one, did you also disable CONFIG_SMP in your Linux .config?

In Yocto, you would edit meta-polarfire-soc-yocto-bsp/recipes-kernel/linux/files/icicle-kit-es-sd/defconfig

Is it possible to provide your WIC image so that I can try to debug this a bit further?

Hello @ivan.griffin,

Thank you for your reply, and sorry for my late reply.

I have tried Linux kernel without CONFIG_SMP as you suggested and the following hss-payload-generator config.

set-name: 'PolarFire-SoC-HSS::1217-1340'

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x08080000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  my_baremetal.elf: {
    exec-addr: '0x08080000',
    owner-hart: u54_2,
    secondary-hart: u54_3,
    secondary-hart: u54_4,
    priv-mode: prv_m
  }
  polarfire-soc-buildroot-sdk/work/u-boot-smode/u-boot.bin: {
    exec-addr: '0x80200000',
    owner-hart: u54_1,
    priv-mode: prv_s
  }

It worked fine. Linux booted successfully and I saw login prompt. Bare-metal code looks working at the address 0x0808_0000 as expected. lscpu showed the following.

# lscpu 
Architecture:        riscv64
Byte Order:          Little Endian
CPU(s):              1
On-line CPU(s) list: 0
Thread(s) per core:  1
Core(s) per socket:  1
Socket(s):           1
L1i cache:           16 KiB

However, configuration for “2 harts for Linux-SMP and 2 harts for bare-metal” still doesn’t work… In the case, I defined CONFIG_SMP again because I need 2 harts for Linux kernel. Do you agree on this Linux config??

I put the WIC I tried below. Please download it.

https://flogics-misc.s3.amazonaws.com/20201217/wic_12171455.img.gz

In addition, I will show the patch to polarfire-soc-buildroot-sdk (commit 6514a11) below.

diff --git a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
index 8443d55..9052179 100644
--- a/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
+++ b/conf/icicle-kit-es-sd/icicle-kit-es-sd.dts
@@ -92,7 +92,7 @@
                         reg = <3>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L17: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
@@ -117,7 +117,7 @@
                         reg = <4>;
                         riscv,isa = "rv64imafdc";                        
                         tlb-split;
-                        status = "okay";
+                        status = "disabled";
                        L21: interrupt-controller {
                                #interrupt-cells = <1>;
                                compatible = "riscv,cpu-intc";
@@ -322,4 +322,4 @@
                reg = <0x20 0x30000000 0 0x80000000 >;
                 status = "okay";
         };
-};
\ No newline at end of file
+};
diff --git a/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch b/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
index afef581..2c035f0 100644
--- a/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
+++ b/patches/u-boot/0003-device-tree-for-the-Microchip-mpfs-icicle-SoC-board.patch
@@ -135,7 +135,7 @@ index 0000000000..29af1f82ba
 +                      reg = <3>;
 +                      riscv,isa = "rv64imafdc";
 +                      sifive,itim = <&L18>;
-+                      status = "okay";
++                      status = "disabled";
 +                      tlb-split;
 +                      L17: interrupt-controller {
 +                              #interrupt-cells = <1>;
@@ -162,7 +162,7 @@ index 0000000000..29af1f82ba
 +                      reg = <4>;
 +                      riscv,isa = "rv64imafdc";
 +                      sifive,itim = <&L22>;
-+                      status = "okay";
++                      status = "disabled";
 +                      tlb-split;
 +                      L21: interrupt-controller {
 +                              #interrupt-cells = <1>;

Used hss-payload-generator config is as following.

set-name: 'PolarFire-SoC-HSS::1217-1439'

hart-entry-points: {
  u54_1: '0x80200000',
  u54_2: '0x80200000',
  u54_3: '0x08080000',
  u54_4: '0x08080000'
}

payloads:
  my_baremetal.elf: {
    exec-addr: '0x08080000',
    owner-hart: u54_3,
    secondary-hart: u54_4,
    priv-mode: prv_m
  }
  polarfire-soc-buildroot-sdk/work/u-boot-smode/u-boot.bin: {
    exec-addr: '0x80200000',
    owner-hart: u54_1,
    secondary-hart: u54_2,
    priv-mode: prv_s
  }

Linux boot message is as following. (Looks as same as the last trial in November.)

[    0.776666] Rounding down aligned max_sectors from 4294967295 to 4294967288
[    0.784246] db_root: cannot open: /etc/target
[    0.789636] libphy: Fixed MDIO Bus: probed
[    0.795617] libphy: MACB_mii_bus: probed
[   21.830462] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[   21.836413] rcu:     1-...0: (1 GPs behind) idle=792/1/0x4000000000000000 softirq=31/32 fqs=1050
[   21.845037]  (detected by 0, t=2102 jiffies, g=-1111, q=2)
[   21.850526] Task dump for CPU 1:
[   21.853767] modprobe        R  running task        0    35      7 0x00000008
[   21.860830] Call Trace:
[   21.863310] [<ffffffe00002e5f4>] ret_from_kernel_thread+0x0/0x10

Once again, any suggestion would be appreciated.

Regards,
Atsushi