[fedora-arm] Re: Latest Fedora update leaves aarch64 unbootable RPi V3.

On Wed, 2020-02-19 at 11:25 -0500, Paul Whalen wrote:
> Hi Michael,

> Trying to reproduce this issue without success. How did you write the
> images and
> what command was used? After booting, did you confirm the kargs were
> as expected?

I used arm-install as recommended.

> Once booted, you just 'dnf update' and get the corrupted grub.cfg?

Correct.

One of the systems required a reinstall of kernel-core-5.4.19-
200.fc31.aarch64 in order to fix a missing initramfs (the system had
crashed).

Saw this in the subsequent the reinstall and the grub.cfg that started
out as 5973 bytes was now 841768 bytes with corrupted
"default_kernelopts=" in the file. Fortunately, I had backed up the
grub.cfg and was able to restored it before trying to reboot the
system.

--
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Reinstalling : kernel-core-5.4.19-200.fc31.aarch64 1/2
Running scriptlet: kernel-core-5.4.19-200.fc31.aarch64 1/2
Running scriptlet: kernel-core-5.4.19-200.fc31.aarch64 2/2
Cleanup : kernel-core-5.4.19-200.fc31.aarch64 2/2
Running scriptlet: kernel-core-5.4.19-200.fc31.aarch64 2/2
grubby fatal error: unable to find a suitable template

Verifying : kernel-core-5.4.19-200.fc31.aarch64 1/2
Verifying : kernel-core-5.4.19-200.fc31.aarch64 2/2
Completion plugin: Generating completion cache...

Reinstalled:
kernel-core-5.4.19-200.fc31.aarch64

Complete!
--

Now have a bug report in Bugzilla and working it. Just attached a
stack of files requested to the bug report.

https://bugzilla.redhat.com/show_bug.cgi?id=1804483

Mike

> Thanks!
> Paul
>
>
> ----- Original Message -----
> > On Tue, 2020-02-18 at 20:22 -0500, Michael H. Warfield wrote:
> > > On Tue, 2020-02-18 at 19:44 -0500, Stuart D. Gathman wrote:
> > > > On Tue, 18 Feb 2020, Michael H. Warfield wrote:
> > > >
> > > > > So the problem is in grubby. This had to have happened in
> > > > > just
> > > > > the
> > > > > last week. The update to 5.4.18 did not result in this
> > > > > carnage. So,
> > > > > whatever happened with grubby happened recently.
> > > > > Strangely, the /boot/efi/EFI/fedora/grub.cfg file is in the
> > > > > aarch64
> > > > > image but not in the arm7l image. :-?
> > > > It seems like this bug would affect all EFI systems, not just
> > > > aarch64.
> > > > I'm be on the alert before installing 5.4.19 on x86_64 as well.
> > > I agree. Seems strange but...
> > > I'm actually wondering if the hang and crash is a buffer
> > > overrun. I
> > > looked on a newly rebuilt system and saw the corruption but it
> > > was
> > > only
> > > a few lines long and the system still worked. I've got a pile of
> > > development systems are updated every time a kernel gets updated
> > > (once
> > > or twice a week) that that line had hit over 800K long. They're
> > > all
> > > largely mirrors of each other, just with different tasks. Each
> > > of
> > > the
> > > "corrupt" files were identical but all of the updates have been
> > > in
> > > lock
> > > step.
> > > All but one of the six affected systems are back up and the
> > > single
> > > one
> > > that's not managed to boot the kernel and started but ran into a
> > > panic.
> > > But I get a kernel menu on it now.
> >
> > That's it...
> >
> > The panic on the odd system out was a missing initramfs but that's
> > one
> > of the two that hurled chunks and crashed during the update and I
> > got
> > it back on the previous kernel and reinstalled the newer kernel to
> > fix
> > that once I had fixed grub.cfg. I then compared the working
> > grub.cfg
> > file to the resulting one from the reinstall.
> >
> > This:
> > --
> > set default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40 ro
> > cma=192MB"
> > --
> > Became this during the install:
> > --
> > set
> > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40
> > ro cma=192MB"
> > --
> >
> > And became this after the reinstall was complete:
> > --
> > set
> > default_kernelopts="root=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-
> > b3d2-494a-b4ec-ebb4687d6b40=UUID=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40=92567a3e-b3d2-
> > 494a-b4ec-ebb4687d6b40=92567a3e-b3d2-494a-b4ec-
> > ebb4687d6b40=92567a3e-b3d2-494a-b4ec-ebb4687d6b40
> > ro cma=192MB"
> > --
> >
> > It duplicated the root UUID on the default_kernelopts several
> > times.
> > And rebooting with that bad line still worked. It seems to be the
> > size
> > of that duplicated root UUID that finally borks it. Each time the
> > kernel gets updated, it gets another string attached until it blows
> > chunks. This may have been in there for a long time, I'm just real
> > aggressive about doing kernel updates.
> >
> > It's now up on the updated kernel. WIERD. I should probably
> > update my
> > Bugzilla report now.
> >
> > No idea why it's not showing up on x86_64 systems.
> >
> > > Just checked my x86_64 system (a Lenovo Yoga 730-15). No sign of
> > > the
> > > corruption in that file. Very strange but 6 systems impacted at
> > > nearly
> > > the same time (hours).
> >
> > Mike
> > --
> > Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 |
> > mhw@WittsEnd.com
> > /\/\|=mhw=|\/\/ | (c) +1 678 463-0932 |
> > http://www.wittsend.com/mhw/
> > ARIN whois: MHW9-ARIN | An optimist believes we live in the
> > best of all
> > PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure
> > of it!
> >
> > _______________________________________________
> > arm mailing list -- arm@lists.fedoraproject.org
> > To unsubscribe send an email to arm-leave@lists.fedoraproject.org
> > Fedora Code of Conduct:
> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> > List Guidelines:
> > https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/arm@lists.fedoraproject.org
> >
>
>
--
Michael H. Warfield (AI4NB) | (o) +1 706 850-8773 | mhw@WittsEnd.com
/\/\|=mhw=|\/\/ | (c) +1 678 463-0932 | http://www.wittsend.com/mhw/
ARIN whois: MHW9-ARIN | An optimist believes we live in the best of all
PGP Key: 0xC0EB9675674627FF | possible worlds. A pessimist is sure of it!