[fedora-arm] Re: Latest Fedora update leaves aarch64 unbootable RPi V3.

On Wed, 2020-02-19 at 11:56 -0500, Michael H. Warfield wrote:
> On Wed, 2020-02-19 at 11:25 -0500, Paul Whalen wrote:
> > Hi Michael,
> > Trying to reproduce this issue without success. How did you write
> > the
> > images and
> > what command was used? After booting, did you confirm the kargs
> > were
> > as expected?

> I used arm-install as recommended.

Specific command I used was...

arm-image-installer --image=Fedora-Xfce-31-1.9.aarch64.raw.xz --target=rpi3 --selinux=off --norootpass --resizefs --addconsole

With some keys options and output.

> > Once booted, you just 'dnf update' and get the corrupted grub.cfg?
>
> Correct.
>
> One of the systems required a reinstall of kernel-core-5.4.19-
> 200.fc31.aarch64 in order to fix a missing initramfs (the system had
> crashed).
>
> Saw this in the subsequent the reinstall and the grub.cfg that
> started
> out as 5973 bytes was now 841768 bytes with corrupted
> "default_kernelopts=" in the file. Fortunately, I had backed up the
> grub.cfg and was able to restored it before trying to reboot the
> system.
>
> --
> Running transaction check
> Transaction check succeeded.
> Running transaction test
> Transaction test succeeded.
> Running transaction
>
> Preparing :
> 1/1
> Reinstalling : kernel-core-5.4.19-
> 200.fc31.aarch64 1/2
> Running scriptlet: kernel-core-5.4.19-
> 200.fc31.aarch64 1/2
> Running scriptlet: kernel-core-5.4.19-
> 200.fc31.aarch64 2/2
> Cleanup : kernel-core-5.4.19-
> 200.fc31.aarch64 2/2
> Running scriptlet: kernel-core-5.4.19-
> 200.fc31.aarch64 2/2
> grubby fatal error: unable to find a suitable template
>
> Verifying : kernel-core-5.4.19-
> 200.fc31.aarch64 1/2
> Verifying : kernel-core-5.4.19-
> 200.fc31.aarch64 2/2
> Completion plugin: Generating completion cache...
>
> Reinstalled:
> kernel-core-5.4.19-
> 200.fc31.aarch64
>
> Complete!
> --
>
> Now have a bug report in Bugzilla and working it. Just attached a
> stack of files requested to the bug report.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1804483
>
> Mike
>
> > Thanks!
> > Paul
> >
> >
> > ----- Original Message -----
> > > On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> > > > On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > > > > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> > > > >
> > > > > > So the problem is in grubby. This had to have happened in
> > > > > > just
> > > > > > the
> > > > > > last week. The update to 5.4.18 did not result in this
> > > > > > carnage. So,
> > > > > > whatever happened with grubby happened recently.
> > > > > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in the
> > > > > > aarch64
> > > > > > image but not in the arm7l image. :-?
> > > > > It seems like this bug would affect all EFI systems, not just
> > > > > aarch64.
> > > > > I'm be on the alert before installing 5.4.19 on x86_64 as
> > > > > well.
> > > > I agree. Seems strange but...
> > > > I'm actually wondering if the hang and crash is a buffer
> > > > overrun. I
> > > > looked on a newly rebuilt system and saw the corruption but it
> > > > was
> > > > only
> > > > a few lines long and the system still worked. I've got a pile
> > > > of
> > > > development systems are updated every time a kernel gets
> > > > updated
> > > > (once
> > > > or twice a week) that that line had hit over 800K
> > > > long. They're
> > > > all
> > > > largely mirrors of each other, just with different tasks. Each
> > > > of
> > > > the
> > > > "corrupt" files were identical but all of the updates have been
> > > > in
> > > > lock
> > > > step.
> > > > All but one of the six affected systems are back up and the
> > > > single
> > > > one
> > > > that's not managed to boot the kernel and started but ran into
> > > > a
> > > > panic.
> > > > But I get a kernel menu on it now.
> > >
> > > That's it...
> > >
> > > The panic on the odd system out was a missing initramfs but
> > > that's
> > > one
> > > of the two that hurled chunks and crashed during the update and I
> > > got
> > > it back on the previous kernel and reinstalled the newer kernel
> > > to
> > > fix
> > > that once I had fixed grub.cfg. I then compared the working
> > > grub.cfg
> > > file to the resulting one from the reinstall.
> > >
> > > This:
> > > --
> > > set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40 ro
> > > cma=192MB"
> > > --
> > > Became this during the install:
> > > --
> > > set
> > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > b3d2-494a-b4ec-ebb4687d6b40
> > > ro cma=192MB"
> > > --
> > >
> > > And became this after the reinstall was complete:
> > > --
> > > set
> > > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-
> > > 494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40
> > > ro cma=192MB"
> > > --
> > >
> > > It duplicated the root UUID on the default_kernelopts several
> > > times.
> > > And rebooting with that bad line still worked. It seems to be
> > > the
> > > size
> > > of that duplicated root UUID that finally borks it. Each time
> > > the
> > > kernel gets updated, it gets another string attached until it
> > > blows
> > > chunks. This may have been in there for a long time, I'm just
> > > real
> > > aggressive about doing kernel updates.
> > >
> > > It's now up on the updated kernel. WIERD. I should probably
> > > update my
> > > Bugzilla report now.
> > >
> > > No idea why it's not showing up on x86_64 systems.
> > >
> > > > Just checked my x86_64 system (a Lenovo Yoga 730-15). No sign
> > > > of
> > > > the
> > > > corruption in that file. Very strange but 6 systems impacted
> > > > at
> > > > nearly
> > > > the same time (hours).
> > >
> > > Mike
> > > --
> > > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > > mhw@WittsEnd.com
> > > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > > http://www.wittsend.com/mhw/
> > > ARIN whois: MHW9-ARIN | An optimist believes we live in the
> > > best of all
> > > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is
> > > sure
> > > of it!
> > >
> > > _______________________________________________
> > > arm mailing list -- arm@lists.fedoraproject.org
> > > To unsubscribe send an email to arm-leave@lists.fedoraproject.org
> > > Fedora Code of Conduct:
> > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > > List Guidelines:
> > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> > >
--
Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
/\/\|=mhw=|\/\/ | (c) +1 678 463-0932 | http://www.wittsend.com/mhw/
ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!