[fedora-arm] Boot AARCH64 PXE images on Raspberry Pi 3B+/4B

Hey ARM people,

A while back I was working on netbooting Fedora 30 on the Raspberry Pi 3B+ and I eventually got it working by TFTP loading U-Boot first, then loading GRUB2 as the EFI executable and then having GRUB2 load vmlinuz and initrd.img to get the kickstart install going.

The same sequence currently doesn't work on the RPI 4. I'm using a 4GB Rev 1.2 RPI4B:

GRUB2 can't load vmlinuz or initrd.img over HTTP. Taking a tcpdump of the connection, there is a ton of un-acked as well as retried packets in the dump. Loading the images via TFTP works but when GRUB2 is loading them the whole system just freezes. No console output no DHCP discovers getting to my DHCP server, no nothing.

So I started working towards removing GRUB2 and U-Boot from the process. I'm using the kernel + initrd from here: https://dl.fedoraproject.org/pub/fedora/linux/development/32/Everything/aarch64/os/images/pxeboot/

Initially start4x.elf wasn't able to boot vmlinuz at all but then I decompressed the gzipped vmlinuz and the kernel started loading.
The next issue was that the default initrd.img file seems to be too large for the RPI. When downloading, it's about 69MB in size but the kernel seems to only see ~64MB:

[ 0.869951] Trying to unpack rootfs image as initramfs...
[ 23.568924] Initramfs unpacking failed: XZ-compressed data is corrupt
[ 23.626382] Freeing initrd memory: 65532K

I de-compressed the image and re-compressed it on XZ level 9 and managed squish it down to 63MB which got me further.

[ 0.873159] Trying to unpack rootfs image as initramfs...
[ 22.872413] Freeing initrd memory: 63896K

Now the kernel is loading all the way until it's trying to load the stage2 image install.img as well as the kickstart script from network but gets a dracut timeout.

I'm seeing a couple of errors in the DMESG output that hint at some important things going wrong:

[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[ 0.000000] Linux version 5.6.2-301.fc32.aarch64 (mockbuild@buildvm-aarch64-07.arm.fedoraproject.org) (gcc version 10.0.1 20200328 (Red Hat 10.0.1-0.11) (GCC)) #1 SMP Tue Apr 7 18:10:18 UTC 2020
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.2
[ 0.000000] efi: Getting EFI parameters from FDT:
[ 0.000000] efi: UEFI not found.
<... snip ...>
[ 0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=1 cma=64M cma=96M smsc95xx.macaddr=DC:A6:32:80:95:FC vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 console=ttyS0,115200 console=tty1 inst.stage2="http://server.domain.local/boot/fedora/32/aarch64" inst.ks="http://server.domain.local/kickstart/rpi.ks"
<... snip ...>
[ 0.069060] DMI not present or invalid.
<... snip ...>
[ 24.757039] uart-pl011 fe201000.serial: there is not valid maps for state default
[ 24.764885] fe201000.serial: ttyAMA1 at MMIO 0xfe201000 (irq = 20, base_baud = 0) is a PL011 rev2
[ 24.776307] raspberrypi-firmware soc:firmware: Request 0x00000001 returned status 0x00000000
[ 24.784927] raspberrypi-firmware soc:firmware: Request 0x00030046 returned status 0x00000000
[ 24.793836] raspberrypi-firmware soc:firmware: Request 0x00030007 returned status 0x00000000
[ 24.802423] raspberrypi-clk raspberrypi-clk: Failed to get pllb min freq: -22
[ 24.809666] raspberrypi-clk raspberrypi-clk: Failed to initialize pllb, -22
[ 24.816755] raspberrypi-clk: probe of raspberrypi-clk failed with error -22
[ 24.825200] raspberrypi-firmware soc:firmware: Request 0x00030030 returned status 0x00000000
<... snip ...>
[ 36.070243] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 0 config (-22 80)
[ 36.179880] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 1 config (-22 81)
[ 36.248250] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 6 config (-22 86)
[ 36.278132] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 6 config (-22 86)
[ 36.286692] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 2 config (-22 82)
[ 36.352309] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
[ 36.377543] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
[ 36.398166] gpio-regulator: probe of sd_io_1v8_reg failed with error -22
[ 36.405072] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 3 config (-22 83)
[ 36.600708] raspberrypi-exp-gpio soc:firmware:gpio: Failed to get GPIO 4 config (-22 84)
<... snip ...>
[ 36.648764] pci 0000:01:00.0: Failed to load VL805's firmware: -22. Will continue to attempt to work, but bad things might happen. You should fix this...
<... snip ...>
[ 36.880911] xhci_hcd 0000:01:00.0: xHCI Host Controller
[ 36.888039] raspberrypi-firmware soc:firmware: Request 0x00028001 returned status 0x00000000
[ 36.911668] xhci_hcd 0000:01:00.0: new USB bus registered, assigned bus number 2
[ 36.919252] xhci_hcd 0000:01:00.0: Host supports USB 3.0 SuperSpeed
[ 36.925791] xhci_hcd 0000:01:00.0: WARNING: Host System Error
<... snip ...>
[ 39.311759] random: crng init done
[ 39.315229] random: 7 urandom warning(s) missed due to ratelimiting
[ 207.308822] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 208.087573] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 208.788905] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
[ 209.497727] dracut-initqueue[980]: Warning: dracut-initqueue timeout - starting timeout scripts
<.....snip.....>
[ 291.298611] dracut-initqueue[980]: Warning: Could not boot.
Starting Dracut Emergency Shell...

Generating "/run/initramfs/rdsosreport.txt"

Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

I tried using DTBs and firmware from the official Raspberry Pi repository as well as the ones from the boot volume in the aarch64 Fedora Minimal Spin image. Neither made a difference.

So I have 3 questions at this point:
1. Is anyone aware of the 64MB initramfs limit that I'm seeing? Anyway to get around that? This doesn't seem to be an issue in the x86 architecture.
2. What's the story around U-Boot? What is it providing to the kernel that the RPI bootloader can't or doesn't provide?
3. Kernel 5.6 should have support for VL805 chip. Why does it fail loading it?

Thanks guys!
_______________________________________________
arm mailing list -- arm@lists.fedoraproject.org
To unsubscribe send an email to arm-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org